Qubes agents with Archlinux HVM (qrexec - gui ...)

[Olivier Médoc]

Hello,

I finally managed to make my Archlinux HVM working smoothly with Qubes,
even if there is still some issues (discussed here). You can find below
my notes and discovered bugs related to installing qubes agents in a non
Fedora/RH HVM.

- By the time you read this email, I will probably have uploaded in the
AUR archlinux PKGBUILD files for building the necessary packages:
https://aur.archlinux.org/packages/?O=0&K=qubes

- I basically tried to copy the .spec files found in the qubes marmarek
git and converted it to PKGBUILD so that archlinux packages can be built
smoothly

---------------------------------------
---- qubes-vm-xen
---------------------------------------
- qubes-vm-xen is the first required package and allows building xen
core libraries (essentially xenstore stuff and python libraries ?). This
package does not provides systemd files required for xen. As a
consequence, I added proc-xen.mount (used to mount /proc/xen system),
xenconsoled.service, and xenstored.service.

- In the end, I discovered that the only requirement is to add /proc/xen
to fstab, and that xenconsoled or xenstored services are not required. I
modify fstab accordingly during qubes-vm-core package installation.

---------------------------------------
---- qubes-vm-core
---------------------------------------
- qubes-vm-core is the most complicated package as it glue everything
toghether. It installs includes, qubes specific libraries, configuration
files for loading xen and qubes kernel modules...

- most of these files are adapted to a template vm but will break your
HVM (such as misc/fstab ...), so I tried to install as little config
files as possible.

- archlinux now use systemd but the files in /etc/sysconfig are not
exactly the same than in RedHat/Fedora (I hate this stuff anyway). So I
needed to convert module loading stuff. By the way, I added u2mfn
loading in qubes-vm-core instead of qubes-vm-gui because it is required
for qrexec_agent to work correctly.

- Several .sh scripts are using absolute path, which breaks a lot of
things especially in qubes network scripts (archlinux does not use /bin
or /sbin for binaries such as ifconfig). As a consequence, I used sed to
convert it before packaging and removed paths (it seems that $PATH is
used when the scripts are started, that the case for ifconfig, route,
ethtool, ip, grep).

- All required qubes service files are enabled during qubes-vm-core
package installation. I only keep getty.target.wants content because I
will not be able to logon to my HVM anymore. During this process, the
user 'user' is also created in the HVM, because dom0 need it.

---------------------------------------
---- qubes-vm-kernel-modules
---------------------------------------
- The last piece of code required to have the qrexec agent working is
the package qubes-vm-kernel-modules. This package will only build
u2mfn.ko from marmarek kernel git.

- I will create another package for the lfs kernel:
qubes-vm-kernel-modules-lfs

---------------------------------------
---- Enabling qrexec
---------------------------------------
- Now enable qrexec in dom0 for your hvm by using qvm-prefs -s
[yourHVMname] qrexec_installed True. Reboot your hvm, and you will be
able to use:

* qvm-run --pass-io [yourHVMname] "pwd" to run commands or verify that
qrexec is working correctly
* qvm-copy-to-vm from your HVM to copy files to another VM (very useful)
* /usr/lib/qubes/qubes_trigger_sync_appmenus.sh to populate the
application menu in Qubes VM Manager
* the network of your HVM will be setup correctly during HVM startup
* meminfo-writer can be activated through Qubes-VM-manager. But I don't
know it purpose, and it seems to be sometimes disabled or 'forgotten' be
qubes-vm-manager. It seems to be a bug, but I don't know how to
reproduce it at will.

---------------------------------------
---- qubes-vm-gui
---------------------------------------
- The last part concerns the qubes-vm-gui package this package install
the gui agent, with the necessary scripts, xorg modules, and service files.

- Again /etc/sysconfig/desktop is not supported by archlinux. I tried to
run qubes-session through a xinit.d script, but I don't know yet if it
work (this allow running qubes-session which does not look critical).

- The package install process enable qubes-gui-agent. If you reboot, the
agent will be enabled on startup.

- qubes-gui-agent will wait for a dom0 command (qubes_guid). When this
command is received, it will start xorg on the DISPLAY=:0

---------------------------------------
---- Enabling gui agent
---------------------------------------
- Enabling the gui agent does not seems to be possible through standard
tools for HVM VMs. But you can start qubes_guid in dom0 manually using
qubes_guid -d [yourHVMid] -c red -i red -l 1

- You can find your HVM id by running 'xl list' in dom0. Select
'yourHVMname' instead of 'yourHVMname-dm', which seems to be the qemu gui.

- You can now start a graphical program using qvm-run yourHVMname
'export DISPLAY=:0 && terminal' (using --pass-io for debugging)

- I don't know why DISPLAY is not set automatically in the VM. It seems
that inside the HVM qrexec does not load profile.d ?

- It does not seems to be easy to run qubes_guid automatically. The
python code apparently redefines gui startup for HVMs to run through
myHVMname-dm. I can try to write a quick hack, but it will probably not
be what the devs want...

- I found that xenstore-ls store qubes_keyboard only in 'myHVMname-dm'
instead of 'myHVMname'. As a consequence, the keyboard language is not
propagated to my HVM. I did not found where in the code xen store
variables are defined.

- It is possible to hide the gui used to boot the HVM by running
qvm-start --no-guid yourHVMname. But it will appears again when using
qvm-run.

- Another unrelated bug is in
/usr/lib/qubes/network-manager-prepare-conf-dir (in default NetMV):
test -d verify that /etc/NetworkManager/system-connections is a
directory, but it later replace it with a symlink. test -h
/etc/NetworkManager/system-connections should also be used to verify if
this script has already been started, which is the case if I restart
NetworkManager.

[Marek Marczykowski]

On 06.01.2013 18:37, Olivier Médoc wrote:
> Hello,
>
> I finally managed to make my Archlinux HVM working smoothly with Qubes, even
> if there is still some issues (discussed here). You can find below my notes
> and discovered bugs related to installing qubes agents in a non Fedora/RH HVM.
>
> - By the time you read this email, I will probably have uploaded in the AUR
> archlinux PKGBUILD files for building the necessary packages:
> https://aur.archlinux.org/packages/?O=0&K=qubes

I suppose I've read this email to early ;)

> - I basically tried to copy the .spec files found in the qubes marmarek git
> and converted it to PKGBUILD so that archlinux packages can be built smoothly

Great work!

Just general note that linux agents are prepared to work in PV domain, not
HVM. I'm very surprised that you had so little problems in running them under
HVM. Anyway PV should give you some better performance.

To convert your HVM linux domain to PV, you can place system partition alone
(without partition table) in root.img and use it in some
standalonevm/templatevm (use qvm-clone to create new templatevm and override
its root.img).
More general solution would be to prepare template-builder for Archlinux,
which should be quite simple, based on template-builder for Fedora.

This will also make your VM using the same kernel as other VMs (including
/lib/modules, which is mounted in initramfs), so if you need some other kernel
version/config you need to prepare dom0 package with it.

>
> ---------------------------------------
> ---- qubes-vm-xen
> ---------------------------------------
> - qubes-vm-xen is the first required package and allows building xen core
> libraries (essentially xenstore stuff and python libraries ?). This package
> does not provides systemd files required for xen. As a consequence, I added
> proc-xen.mount (used to mount /proc/xen system), xenconsoled.service, and
> xenstored.service.
>
> - In the end, I discovered that the only requirement is to add /proc/xen to
> fstab, and that xenconsoled or xenstored services are not required. I modify
> fstab accordingly during qubes-vm-core package installation.

Yes, xenstored and xenconsoled services are used only in dom0.

> ---------------------------------------
> ---- qubes-vm-core
> ---------------------------------------
> - qubes-vm-core is the most complicated package as it glue everything
> toghether. It installs includes, qubes specific libraries, configuration files
> for loading xen and qubes kernel modules...
>
> - most of these files are adapted to a template vm but will break your HVM
> (such as misc/fstab ...), so I tried to install as little config files as
> possible.

Some of them are not strictly required, but are used to slim down VM.

> - archlinux now use systemd but the files in /etc/sysconfig are not exactly
> the same than in RedHat/Fedora (I hate this stuff anyway). So I needed to
> convert module loading stuff. By the way, I added u2mfn loading in
> qubes-vm-core instead of qubes-vm-gui because it is required for qrexec_agent
> to work correctly.
>
> - Several .sh scripts are using absolute path, which breaks a lot of things
> especially in qubes network scripts (archlinux does not use /bin or /sbin for
> binaries such as ifconfig). As a consequence, I used sed to convert it before
> packaging and removed paths (it seems that $PATH is used when the scripts are
> started, that the case for ifconfig, route, ethtool, ip, grep).

Does Archlinux not have /bin->/usr/bin symlink (Fedora 17 have one)?

meminfo-writer is used to inform dom0 about memory usage inside VM, basically
for dynamic memory management. Don't know if xen-balloon driver works in HVM,
but if yes it should work as soon as you start meminfo-writer in VM (it is
part of qubes-vm-core package). Most likely you also want to raise maxmem
parameter in VM settings for that case.

> ---------------------------------------
> ---- qubes-vm-gui
> ---------------------------------------
> - The last part concerns the qubes-vm-gui package this package install the gui
> agent, with the necessary scripts, xorg modules, and service files.
>
> - Again /etc/sysconfig/desktop is not supported by archlinux. I tried to run
> qubes-session through a xinit.d script, but I don't know yet if it work (this
> allow running qubes-session which does not look critical).
>
> - The package install process enable qubes-gui-agent. If you reboot, the agent
> will be enabled on startup.
>
> - qubes-gui-agent will wait for a dom0 command (qubes_guid). When this command
> is received, it will start xorg on the DISPLAY=:0

qubes-session is supposed to run by xinit just after Xorg start. Don't know
where to configure it - on fedora it was /etc/sysconfig/desktop.

>
> ---------------------------------------
> ---- Enabling gui agent
> ---------------------------------------
> - Enabling the gui agent does not seems to be possible through standard tools
> for HVM VMs. But you can start qubes_guid in dom0 manually using qubes_guid -d
> [yourHVMid] -c red -i red -l 1

For HVM guid is started only for stubdom (where is emulated VGA), because we
don't have gui-agents for any HVM yet.
The code responsible for it is dom0:
http://git.qubes-os.org/?p=marmarek/core.git;a=blob;f=dom0/qvm-core/qubes.py;h=8ef0744cdc7c639ff66b5cbfbecb90292e040a1b;hb=HEAD#l2465

If you convert your domain to PV, it will be automatically "fixed". PV version
of "start_guid" function also waits for qubes-session to finish (to not start
programs before all session-related things will be started up).

> - You can find your HVM id by running 'xl list' in dom0. Select 'yourHVMname'
> instead of 'yourHVMname-dm', which seems to be the qemu gui.
>
> - You can now start a graphical program using qvm-run yourHVMname 'export
> DISPLAY=:0 && terminal' (using --pass-io for debugging)
>
> - I don't know why DISPLAY is not set automatically in the VM. It seems that
> inside the HVM qrexec does not load profile.d ?

qrexec uses "su" to load switch user and load environment - perhaps it is your
pam.d/su settings?

> - It does not seems to be easy to run qubes_guid automatically. The python
> code apparently redefines gui startup for HVMs to run through myHVMname-dm. I
> can try to write a quick hack, but it will probably not be what the devs want...
>
> - I found that xenstore-ls store qubes_keyboard only in 'myHVMname-dm' instead
> of 'myHVMname'. As a consequence, the keyboard language is not propagated to
> my HVM. I did not found where in the code xen store variables are defined.
>
> - It is possible to hide the gui used to boot the HVM by running qvm-start
> --no-guid yourHVMname. But it will appears again when using qvm-run.

All above we will need to be fix anyway to support Windows VMs with GUI agent
installed (which do not exists yet). This can be done similar to qrexec
(qrexec_installed flag) - second flag like guiagent_installed and if it is set
to true, behave mostly like in PV case (perhaps call the same function via eg
super(QubesHVM, self).start_guid etc). Note that most likely you will need to
start qrexec before guid in that case (_start_guid_first internal flag).

For now you have three options:
- convert your VM to PV
- implement above logic in qubes.py (and sent patches :) )
- dirty hack in qubes.py

> - Another unrelated bug is in /usr/lib/qubes/network-manager-prepare-conf-dir
> (in default NetMV): test -d verify that
> /etc/NetworkManager/system-connections is a directory, but it later replace it
> with a symlink. test -h /etc/NetworkManager/system-connections should also be
> used to verify if this script has already been started, which is the case if I
> restart NetworkManager.

Thanks, will fix it.

--
Best Regards / Pozdrawiam,
Marek Marczykowski
Invisible Things Lab

[Olivier Médoc]

Actually, I ran into a lot of problems. But it seems hard to setup a PV
VM from scratch. I will probably try to convert it now that I have
things working and that I have a better understanding of all your scripts.
I don't know how to deal with the template builder. Archlinux use it own
package format instead of rpms which seems to make things more
complicated to convert (at least in my head).

That's worst for your last point as I need to convert a pacman package
to rpm in order to load it in dom0 :S

No, it has a /lib -> /usr/lib but that's not the case for /bin or /sbin

When I raise Max mem in qubes-vm-manager, the configuration is reset to
initial mem=max mem and meminfo-writer is disabled (even if it is still
running in the VM). I fixed this by patching qubes.py but I don't know
yet if xen-balloon works (I don't know if the driver is in my kernel
anyway).

>> ---------------------------------------
>> ---- qubes-vm-gui
>> ---------------------------------------
>> - The last part concerns the qubes-vm-gui package this package install the gui
>> agent, with the necessary scripts, xorg modules, and service files.
>>
>> - Again /etc/sysconfig/desktop is not supported by archlinux. I tried to run
>> qubes-session through a xinit.d script, but I don't know yet if it work (this
>> allow running qubes-session which does not look critical).
>>
>> - The package install process enable qubes-gui-agent. If you reboot, the agent
>> will be enabled on startup.
>>
>> - qubes-gui-agent will wait for a dom0 command (qubes_guid). When this command
>> is received, it will start xorg on the DISPLAY=:0
> qubes-session is supposed to run by xinit just after Xorg start. Don't know
> where to configure it - on fedora it was /etc/sysconfig/desktop.

It seems to work if I put it in /etc/X11/xinit/xinitrc.d/99-qubes-session.sh

>> ---------------------------------------
>> ---- Enabling gui agent
>> ---------------------------------------
>> - Enabling the gui agent does not seems to be possible through standard tools
>> for HVM VMs. But you can start qubes_guid in dom0 manually using qubes_guid -d
>> [yourHVMid] -c red -i red -l 1
> For HVM guid is started only for stubdom (where is emulated VGA), because we
> don't have gui-agents for any HVM yet.
> The code responsible for it is dom0:
> http://git.qubes-os.org/?p=marmarek/core.git;a=blob;f=dom0/qvm-core/qubes.py;h=8ef0744cdc7c639ff66b5cbfbecb90292e040a1b;hb=HEAD#l2465
>
> If you convert your domain to PV, it will be automatically "fixed". PV version
> of "start_guid" function also waits for qubes-session to finish (to not start
> programs before all session-related things will be started up).
>
>> - You can find your HVM id by running 'xl list' in dom0. Select 'yourHVMname'
>> instead of 'yourHVMname-dm', which seems to be the qemu gui.
>>
>> - You can now start a graphical program using qvm-run yourHVMname 'export
>> DISPLAY=:0 && terminal' (using --pass-io for debugging)
>>
>> - I don't know why DISPLAY is not set automatically in the VM. It seems that
>> inside the HVM qrexec does not load profile.d ?
> qrexec uses "su" to load switch user and load environment - perhaps it is your
> pam.d/su settings?

I tried playing with pam.d/su but I broke my HVM :(. I have to figure
out how to connect to it again. I found that when using a command
through qvm-run, $PATH is only set to /bin;/usr/bin and $DISPLAY is not
set. Apparently, it does not load /etc/profile.d/. If I run a terminal,
it's the same, but using su - in the terminal loads the environment
correctly ...

>
>> - It does not seems to be easy to run qubes_guid automatically. The python
>> code apparently redefines gui startup for HVMs to run through myHVMname-dm. I
>> can try to write a quick hack, but it will probably not be what the devs want...
>>
>> - I found that xenstore-ls store qubes_keyboard only in 'myHVMname-dm' instead
>> of 'myHVMname'. As a consequence, the keyboard language is not propagated to
>> my HVM. I did not found where in the code xen store variables are defined.

I still did not found where is qubes_keyboard set (it does not appears
in qubes.py xenstore related functions).

>> - It is possible to hide the gui used to boot the HVM by running qvm-start
>> --no-guid yourHVMname. But it will appears again when using qvm-run.
> All above we will need to be fix anyway to support Windows VMs with GUI agent
> installed (which do not exists yet). This can be done similar to qrexec
> (qrexec_installed flag) - second flag like guiagent_installed and if it is set
> to true, behave mostly like in PV case (perhaps call the same function via eg
> super(QubesHVM, self).start_guid etc). Note that most likely you will need to
> start qrexec before guid in that case (_start_guid_first internal flag).
>
> For now you have three options:
> - convert your VM to PV
> - implement above logic in qubes.py (and sent patches :) )
> - dirty hack in qubes.py

I implemented the logic in qubes.py and qvm-prefs (see attached
patches), and added a similar thing for meminfo-writer so that I can
extend the HVM maxmem.

That works great (except for the DISPLAY=:0 issue), and the qemu console
is not loaded anymore.

Now my next step is to check for sound and clipboard support, memory
management, and then I will try to create a PV. I'm still affraid with
the kernel things, I will probably just untar the archlinux kernel in
dom0 or something like this. Not sure if it could works...

[Marek Marczykowski]

On 07.01.2013 14:30, Olivier Médoc wrote:
> On 01/06/13 19:59, Marek Marczykowski wrote:

> Actually, I ran into a lot of problems. But it seems hard to setup a PV VM
> from scratch. I will probably try to convert it now that I have things working
> and that I have a better understanding of all your scripts.
> I don't know how to deal with the template builder. Archlinux use it own
> package format instead of rpms which seems to make things more complicated to
> convert (at least in my head).

template-builder have only few distribution-specific things - namely yum calls
(in fedorize_image and qubeize_image). What you need it ability to install
system in some directory (like debootstrap for debian) - if pacman can do
this, you can replace yum calls with pacman (or other tool which can do the
job). You need also place qubes-specific packages somewhere. For fedora it is
automatically done by qubes-builder (it places packages in yum_repo_qubes
dir), but for now you can done it manually or by some simple script (look at
template-builder history - there were such script some time ago, at least R1
had such script).

>
> That's worst for your last point as I need to convert a pacman package to rpm
> in order to load it in dom0 :S

You can adopt qubes kernel (if any change really needed) and build rpm from it
(using qubes-builder, or directly by make rpms in "kernel" dir). It is really
simple. I don't think any Archlinux specific change in kernel is really needed
to start VM. Qubes kernel is almost vanilla one (besides some xen specific
changes).

(...)

>>> ---------------------------------------
>>> ---- Enabling qrexec
>>> ---------------------------------------
>>> - Now enable qrexec in dom0 for your hvm by using qvm-prefs -s [yourHVMname]
>>> qrexec_installed True. Reboot your hvm, and you will be able to use:
>>>
>>> * qvm-run --pass-io [yourHVMname] "pwd" to run commands or verify that qrexec
>>> is working correctly
>>> * qvm-copy-to-vm from your HVM to copy files to another VM (very useful)
>>> * /usr/lib/qubes/qubes_trigger_sync_appmenus.sh to populate the application
>>> menu in Qubes VM Manager
>>> * the network of your HVM will be setup correctly during HVM startup
>>> * meminfo-writer can be activated through Qubes-VM-manager. But I don't know
>>> it purpose, and it seems to be sometimes disabled or 'forgotten' be
>>> qubes-vm-manager. It seems to be a bug, but I don't know how to reproduce it
>>> at will.
>> meminfo-writer is used to inform dom0 about memory usage inside VM, basically
>> for dynamic memory management. Don't know if xen-balloon driver works in HVM,
>> but if yes it should work as soon as you start meminfo-writer in VM (it is
>> part of qubes-vm-core package). Most likely you also want to raise maxmem
>> parameter in VM settings for that case.

> When I raise Max mem in qubes-vm-manager, the configuration is reset to
> initial mem=max mem and meminfo-writer is disabled (even if it is still
> running in the VM). I fixed this by patching qubes.py but I don't know yet if
> xen-balloon works (I don't know if the driver is in my kernel anyway).

You can use vm.services['meminfo-writer'] instead of new memwriter_installed.

>
>>> ---------------------------------------
>>> ---- qubes-vm-gui
>>> ---------------------------------------
>>> - The last part concerns the qubes-vm-gui package this package install the gui
>>> agent, with the necessary scripts, xorg modules, and service files.
>>>
>>> - Again /etc/sysconfig/desktop is not supported by archlinux. I tried to run
>>> qubes-session through a xinit.d script, but I don't know yet if it work (this
>>> allow running qubes-session which does not look critical).
>>>
>>> - The package install process enable qubes-gui-agent. If you reboot, the agent
>>> will be enabled on startup.
>>>
>>> - qubes-gui-agent will wait for a dom0 command (qubes_guid). When this command
>>> is received, it will start xorg on the DISPLAY=:0
>> qubes-session is supposed to run by xinit just after Xorg start. Don't know
>> where to configure it - on fedora it was /etc/sysconfig/desktop.

> It seems to work if I put it in /etc/X11/xinit/xinitrc.d/99-qubes-session.sh

What desktop environment startx run normally in Archlinux? It should be the
same configuration file, just start qubes-session instead of gnome-session or
startkde.

>>> ---------------------------------------
>>> ---- Enabling gui agent
>>> ---------------------------------------
>>> - Enabling the gui agent does not seems to be possible through standard tools
>>> for HVM VMs. But you can start qubes_guid in dom0 manually using qubes_guid -d
>>> [yourHVMid] -c red -i red -l 1
>> For HVM guid is started only for stubdom (where is emulated VGA), because we
>> don't have gui-agents for any HVM yet.
>> The code responsible for it is dom0:
>> http://git.qubes-os.org/?p=marmarek/core.git;a=blob;f=dom0/qvm-core/qubes.py;h=8ef0744cdc7c639ff66b5cbfbecb90292e040a1b;hb=HEAD#l2465
>>
>>
>> If you convert your domain to PV, it will be automatically "fixed". PV version
>> of "start_guid" function also waits for qubes-session to finish (to not start
>> programs before all session-related things will be started up).
>>
>>> - You can find your HVM id by running 'xl list' in dom0. Select 'yourHVMname'
>>> instead of 'yourHVMname-dm', which seems to be the qemu gui.
>>>
>>> - You can now start a graphical program using qvm-run yourHVMname 'export
>>> DISPLAY=:0 && terminal' (using --pass-io for debugging)
>>>
>>> - I don't know why DISPLAY is not set automatically in the VM. It seems that
>>> inside the HVM qrexec does not load profile.d ?
>> qrexec uses "su" to load switch user and load environment - perhaps it is your
>> pam.d/su settings?

> I tried playing with pam.d/su but I broke my HVM :(. I have to figure out how
> to connect to it again. I found that when using a command through qvm-run,
> $PATH is only set to /bin;/usr/bin and $DISPLAY is not set. Apparently, it
> does not load /etc/profile.d/. If I run a terminal, it's the same, but using
> su - in the terminal loads the environment correctly ...

Weird... You can use "strace -f -p `pidof qrexec_agent`" to track this issue
down.

>>
>>> - It does not seems to be easy to run qubes_guid automatically. The python
>>> code apparently redefines gui startup for HVMs to run through myHVMname-dm. I
>>> can try to write a quick hack, but it will probably not be what the devs
>>> want...
>>>
>>> - I found that xenstore-ls store qubes_keyboard only in 'myHVMname-dm' instead
>>> of 'myHVMname'. As a consequence, the keyboard language is not propagated to
>>> my HVM. I did not found where in the code xen store variables are defined.

> I still did not found where is qubes_keyboard set (it does not appears in
> qubes.py xenstore related functions).

It is set by qubes_guid, just before connecting to VM. If you start guid for
VM itself (not only stubdom), it should also create xenstore entry in 'myHVMname'.

>>> - It is possible to hide the gui used to boot the HVM by running qvm-start
>>> --no-guid yourHVMname. But it will appears again when using qvm-run.
>> All above we will need to be fix anyway to support Windows VMs with GUI agent
>> installed (which do not exists yet). This can be done similar to qrexec
>> (qrexec_installed flag) - second flag like guiagent_installed and if it is set
>> to true, behave mostly like in PV case (perhaps call the same function via eg
>> super(QubesHVM, self).start_guid etc). Note that most likely you will need to
>> start qrexec before guid in that case (_start_guid_first internal flag).
>>
>> For now you have three options:
>> - convert your VM to PV
>> - implement above logic in qubes.py (and sent patches :) )
>> - dirty hack in qubes.py

> I implemented the logic in qubes.py and qvm-prefs (see attached patches), and
> added a similar thing for meminfo-writer so that I can extend the HVM maxmem.

As noted earlier, vm.services['meminfo-writer'] could be used instead of
memwriter_installed. Generally vm.services is used to control which services
in VM should start. This is also represented as "enable in memory balancing"
in Qubes VM Manager (but the option is grayed out for HVM...).

Anyway I will look closer at the patch later and maybe apply to qubes code :)

> That works great (except for the DISPLAY=:0 issue), and the qemu console is
> not loaded anymore.
>
> Now my next step is to check for sound and clipboard support, memory
> management, and then I will try to create a PV. I'm still affraid with the
> kernel things, I will probably just untar the archlinux kernel in dom0 or
> something like this. Not sure if it could works...

If you've started guid in VM, clipboard should just work. Sound uses
pulseaudio - in VM should be started with "start-puleaudio-with-vchan" just
after Xorg startup (qubes-gui-vm package used to place
/etc/xdg/autostart/qubes-pulseaudio.desktop file). It needs access to
/proc/u2mfn and /dev/xen/evtchn as normal user.

[Olivier Médoc]

On 01/07/13 15:18, Marek Marczykowski wrote:

On 07.01.2013 14:30, Olivier Médoc wrote:

On 01/06/13 19:59, Marek Marczykowski wrote:

Actually, archlinux policy is "use vanilla if possible, wait for upstream to provide patches". So it should be OK. I will dig into this next week.

Good idea. I modified the code, but I wait to fix other bugs before sending the patch to you ?

---------------------------------------
---- qubes-vm-gui
---------------------------------------
- The last part concerns the qubes-vm-gui package this package install the gui
agent, with the necessary scripts, xorg modules, and service files.

- Again /etc/sysconfig/desktop is not supported by archlinux. I tried to run
qubes-session through a xinit.d script, but I don't know yet if it work (this
allow running qubes-session which does not look critical).

- The package install process enable qubes-gui-agent. If you reboot, the agent
will be enabled on startup.

- qubes-gui-agent will wait for a dom0 command (qubes_guid). When this command
is received, it will start xorg on the DISPLAY=:0

qubes-session is supposed to run by xinit just after Xorg start. Don't know
where to configure it - on fedora it was /etc/sysconfig/desktop.

It seems to work if I put it in /etc/X11/xinit/xinitrc.d/99-qubes-session.sh

What desktop environment startx run normally in Archlinux? It should be the
same configuration file, just start qubes-session instead of gnome-session or
startkde.

Currently, it seems that archlinux uses a .service file for starting the Desktop Manager of your choice (kdm, gdm...), which then run the right command (startxfce, startkde...) from ~/.xinitrc based on input provided by the DM.

Apparently, my hack works if I use /etc/X11/xinit/xinitrc.d/z-qubes-session.sh so that it is started last (copied using qubes copy&paste implementation :) ):

$ cat ~/.xsession-errors

ERROR: ld.so: object 'relaxed_xf86ValidateModes.so' from LD_PRELOAD cannot be preloaded: ignored.

This is a pre-release version of the X server from The X.Org Foundation.
It is not supported in any way.
Bugs may be filed in the bugzilla at http://bugs.freedesktop.org/.
[...]
Initializing built-in extension DRI2
Loading extension GLX
Starting qubes-session...
/usr/bin/qubes-session: ligne47: /usr/bin/gnome-keyring-daemon: Aucun fichier ou dossier de ce type

Please note the ERROR in first line, which seems to be caused because LD_PRELOAD can not be used with setuid executables, which is the case for Xorg:
-rwsr-xr-x 1 root root 2129536 Dec  8 09:38 /usr/bin/Xorg

Is relaxed_xf86ValidateModes.so important ?

I'm convinced that it is a bug in su or in bash (probably related to archlinux choice of linux tools because of GPLv3 being used):
https://bugs.archlinux.org/task/31831

Don't know why they don't want to use GPL3. The workaround is to use zsh for user 'user' instead of bash. Now all environment variables are loaded correctly.

During the debugging, I identified other bugs:

- nogui is not implemented in linux qrexec agent, which makes bash calling 'nogui:echo' -> command not found. This breaks the command echo $$ >> /tmp/qubes-session-waiter which is started for the GUI. Maybe it can simply be inhibited in the qrexec agent code by removing 'nogui:' ?

- I the HVM code, I disabled _start_guid_first when guiagent_installed is True

- QUBESRPC qubes.WaitForSession seems to be started in HVMs. What is the goal of this RPC command ? Moreover, the user 'SYSTEM' is used to start this RPC command, which fails on linux. Should I add 'system_user' in qvm-prefs ?

Thanks a lot for your help.

- It does not seems to be easy to run qubes_guid automatically. The python
code apparently redefines gui startup for HVMs to run through myHVMname-dm. I
can try to write a quick hack, but it will probably not be what the devs
want...

- I found that xenstore-ls store qubes_keyboard only in 'myHVMname-dm' instead
of 'myHVMname'. As a consequence, the keyboard language is not propagated to
my HVM. I did not found where in the code xen store variables are defined.

I still did not found where is qubes_keyboard set (it does not appears in
qubes.py xenstore related functions).

It is set by qubes_guid, just before connecting to VM. If you start guid for
VM itself (not only stubdom), it should also create xenstore entry in 'myHVMname'.

Ok, with the patch, it now appears in the HVM.

- It is possible to hide the gui used to boot the HVM by running qvm-start
--no-guid yourHVMname. But it will appears again when using qvm-run.

All above we will need to be fix anyway to support Windows VMs with GUI agent
installed (which do not exists yet). This can be done similar to qrexec
(qrexec_installed flag) - second flag like guiagent_installed and if it is set
to true, behave mostly like in PV case (perhaps call the same function via eg
super(QubesHVM, self).start_guid etc). Note that most likely you will need to
start qrexec before guid in that case (_start_guid_first internal flag).

For now you have three options:
  - convert your VM to PV
  - implement above logic in qubes.py (and sent patches :) )
  - dirty hack in qubes.py

I implemented the logic in qubes.py and qvm-prefs (see attached patches), and
added a similar thing for meminfo-writer so that I can extend the HVM maxmem.

As noted earlier, vm.services['meminfo-writer'] could be used instead of
memwriter_installed. Generally vm.services is used to control which services
in VM should start. This is also represented as "enable in memory balancing"
in Qubes VM Manager (but the option is grayed out for HVM...).

Anyway I will look closer at the patch later and maybe apply to qubes code :)

That works great (except for the DISPLAY=:0 issue), and the qemu console is
not loaded anymore.
Maybe it can simply be inhibited
Now my next step is to check for sound and clipboard support, memory
management, and then I will try to create a PV. I'm still affraid with the
kernel things, I will probably just untar the archlinux kernel in dom0 or
something like this. Not sure if it could works...

[Marek Marczykowski]

On 08.01.2013 10:27, Olivier Médoc wrote:
> On 01/07/13 15:18, Marek Marczykowski wrote:

>> On 07.01.2013 14:30, Olivier MĆ©doc wrote:
>>> On 01/06/13 19:59, Marek Marczykowski wrote:

If 3.2.30 kernel is ok for you, you probably don't have to do anything.

ok.

> *ERROR: ld.so: object 'relaxed_xf86ValidateModes.so' from LD_PRELOAD cannot be
> preloaded*: ignored.

>
> This is a pre-release version of the X server from The X.Org Foundation.
> It is not supported in any way.
> Bugs may be filed in the bugzilla at http://bugs.freedesktop.org/.
> [...]
> Initializing built-in extension DRI2
> Loading extension GLX
> Starting qubes-session...
> /usr/bin/qubes-session: ligne47: /usr/bin/gnome-keyring-daemon: Aucun fichier
> ou dossier de ce type
>
> Please note the ERROR in first line, which seems to be caused because
> LD_PRELOAD can not be used with setuid executables, which is the case for Xorg:

> -rwsr-xr-x 1 root root 2129536 DecĀ 8 09:38 /usr/bin/Xorg
>
> Is relaxed_xf86ValidateModes.so important ?

This is to allow Xorg using some resolutions, which normally whouldn't be
used. In VM it is important to have the same resolution as in dom0, but other
parameters (refresh rate etc) aren't significant (it uses dummy driver anyway).

The relaxed_xf86ValidateModes.so library should have setuid to be preloaded to
setuid binary.

Do you tried to set /bin/bash as shell instead of /bin/sh?

>
> During the debugging, I identified other bugs:
>
> - nogui is not implemented in linux qrexec agent, which makes bash calling
> 'nogui:echo' -> command not found. This breaks the command echo $$ >>
> /tmp/qubes-session-waiter which is started for the GUI. Maybe it can simply be
> inhibited in the qrexec agent code by removing 'nogui:' ?

Ah, right. This is implemented only in Windows agent, it can be safely ignored
in linux one. Fix pushed.

> - I the HVM code, I disabled _start_guid_first when guiagent_installed is True
>
> - QUBESRPC qubes.WaitForSession seems to be started in HVMs. What is the goal
> of this RPC command ? Moreover, the user 'SYSTEM' is used to start this RPC
> command, which fails on linux. Should I add 'system_user' in qvm-prefs ?

This is equivalent of above "echo $$ >> ...", but done more elegant for
Windows. The problem is that "qubes.WaitForSession" must be stared as
SYSTEM/root on Windows, but as normal user on Linux. Perhaps something like:
su -c "echo $$ >> /tmp/qubes-session-waiter; [ ! -f /tmp/qubes-session-env ]
&& exec sleep 365d" - user
in /etc/qubes_rpc/qubes.WaitForSession (in VM) will solve the problem. Of
course also change user from "SYSTEM" to "root". "root" is automatically
mapped to SYSTEM by windows agent, so it shouldn't break anything.

[Olivier Médoc]

OK I will try that and upload the new PKGBUILD to archlinux aur.

Yes, actually, I was using /bin/bash. Now I will stay with zsh. I don't
wan't to debug archlinux packages (they closed the given ticket anyway
arguing that it is a normal 'old bash' behavior...).

>> During the debugging, I identified other bugs:
>>
>> - nogui is not implemented in linux qrexec agent, which makes bash calling
>> 'nogui:echo' -> command not found. This breaks the command echo $$ >>
>> /tmp/qubes-session-waiter which is started for the GUI. Maybe it can simply be
>> inhibited in the qrexec agent code by removing 'nogui:' ?
> Ah, right. This is implemented only in Windows agent, it can be safely ignored
> in linux one. Fix pushed.
>
>> - I the HVM code, I disabled _start_guid_first when guiagent_installed is True
>>
>> - QUBESRPC qubes.WaitForSession seems to be started in HVMs. What is the goal
>> of this RPC command ? Moreover, the user 'SYSTEM' is used to start this RPC
>> command, which fails on linux. Should I add 'system_user' in qvm-prefs ?
> This is equivalent of above "echo $$ >> ...", but done more elegant for
> Windows. The problem is that "qubes.WaitForSession" must be stared as
> SYSTEM/root on Windows, but as normal user on Linux. Perhaps something like:
> su -c "echo $$ >> /tmp/qubes-session-waiter; [ ! -f /tmp/qubes-session-env ]
> && exec sleep 365d" - user
> in /etc/qubes_rpc/qubes.WaitForSession (in VM) will solve the problem. Of
> course also change user from "SYSTEM" to "root". "root" is automatically
> mapped to SYSTEM by windows agent, so it shouldn't break anything.
>

OK I implemented qubes.WaitForSession and removed linux specific things
from qubes.py (references to qubes-session-waiter). The consequence is
that qubes.WaitForSession should be added to the fedora template-VM or
it won't handle correctly the session waiting mecanism.

I also did not tested it with the windows agent.

Another thing is that I added notify_function references in qrexec in
order to give feedback to the GUI.

[Joanna Rutkowska]

On 01/07/13 15:18, Marek Marczykowski wrote:

/.../

> template-builder have only few distribution-specific things - namely yum calls
> (in fedorize_image and qubeize_image). What you need it ability to install
> system in some directory (like debootstrap for debian) - if pacman can do
> this, you can replace yum calls with pacman (or other tool which can do the
> job). You need also place qubes-specific packages somewhere. For fedora it is
> automatically done by qubes-builder (it places packages in yum_repo_qubes
> dir), but for now you can done it manually or by some simple script (look at
> template-builder history - there were such script some time ago, at least R1
> had such script).
>

I think we should make the whole clean_images/ dir to be distro-specific:

http://git.qubes-os.org/?p=joanna/template-builder.git;a=tree;f=clean_images;h=cfb0cc5c0f21e17d975cffed4d02323b356085a6;hb=HEAD


Perhaps we could just have:
clean_images_fc17
clean_images_fc18
clean_images_archXXX
etc

... and just link to the proper one on build.

j.

[Marek Marczykowski]

This isn't the only distro-specific thing. At least fedorize_image and
qubeize_image assumes that template is Fedora (or at least rpm-based distro)...

[Marek Marczykowski]

All looks good (I haven't tested it yet however). Can you split patches to
individual changes (WaitForSession for linux, guiagent_installed option etc)?
You probably want look here:
http://wiki.qubes-os.org/trac/wiki/DevelopmentWorkflow#RepositoriesandCommittingCode
If you do not want to setup github (or sth) repository, you can use "git
format-patch" and send patches to the list. Sending as ready commits will make
it easy to place you as an author of change in code history.

[Olivier Médoc]

On 01/09/13 02:38, Marek Marczykowski wrote:
>
> All looks good (I haven't tested it yet however). Can you split patches to
> individual changes (WaitForSession for linux, guiagent_installed option etc)?
> You probably want look here:
> http://wiki.qubes-os.org/trac/wiki/DevelopmentWorkflow#RepositoriesandCommittingCode
> If you do not want to setup github (or sth) repository, you can use "git
> format-patch" and send patches to the list. Sending as ready commits will make
> it easy to place you as an author of change in code history.
>

Is it sufficient ? I tried to sign a new git tag, but it doesn't appears
in the patches...

[Marek Marczykowski]

On 09.01.2013 10:23, Olivier Médoc wrote:
> On 01/09/13 02:38, Marek Marczykowski wrote:
>>
>> All looks good (I haven't tested it yet however). Can you split patches to
>> individual changes (WaitForSession for linux, guiagent_installed option etc)?
>> You probably want look here:
>> http://wiki.qubes-os.org/trac/wiki/DevelopmentWorkflow#RepositoriesandCommittingCode
>>
>> If you do not want to setup github (or sth) repository, you can use "git
>> format-patch" and send patches to the list. Sending as ready commits will make
>> it easy to place you as an author of change in code history.
>>
> Is it sufficient ?

The second patch still contains bundled few independent changes - at least
wait_for_session should be in separate patch, but preferable all changes
listed in commit message worth own patch.

> I tried to sign a new git tag, but it doesn't appears in
> the patches...

It is correct - signed tags can be used when you push code to some git server,
they aren't exported to individual patches.

[Olivier Médoc]

Yes, sorry for that, but as I didn't used git in the first place, I
didn't commited all the steps separately. The solution would be to redo
all the different steps manually through git, test it, then regenerate
patches :(

I can still separate things in separate commits, but qubes.py will be
broken if you don't apply all of them.

By the way, how do you test your code ? You are developping directly in
dom0 ? Or using a git somewhere in your dom0 home, linking all the
necessary stuff ?

[Marek Marczykowski]

wait_for_session looks independent of other changes (should work even without
any of your HVM change). So at least split this to separate patch. Splitting
the rest can be a bit tricky now (as you said) so can go as one commit - at
least it all touches HVM code (with exception for notify_function, but it is
bearable).

BTW: useful git commands to refactor commits:
git rebase -i
git add -p

> By the way, how do you test your code ? You are developping directly in dom0 ?
> Or using a git somewhere in your dom0 home, linking all the necessary stuff ?

I have separate QubesVM for development. Depending on complexity of change I
test it on the same machine dom0 (for small changes) - using simple scripts to
retrieve files from the VM (and take backup of original ones...). For bigger
changes I use separate test machine.

Take a look here:
https://wiki.qubes-os.org/trac/wiki/DevelopmentWorkflow

[Olivier Médoc]

I tried to separate all the patches. I hope I didn't introduced any typo
because I worked manually with raw patches...

[Marek Marczykowski]

I've commited the code, thanks! :) (with minor fixes - mostly trailing
whitespaces)

Haven't checked Archlinux HVM with qubes stuff installed, but generic HVM is
still working. Linux PV also (with updated qubes-core-vm).

I have completely no experience with ArchLinux, but the below link looks like
good staring point for archlinux template-builder (if you want to prepare one).

https://wiki.archlinux.org/index.php/Install%20from%20Existing%20Linux#Method_1:_Directly_bootstrapping_Archlinux

About dynamic memory management (not so important now):
xen-balloon doesn't look to be working in HVM at this stage, so enabling
meminfo-writer will do almost nothing. Anyway even if it would work, some
further changes in qubes-core-dom0 are required:
- remove "attrs['maxmem'].pop('save')" line from QubesHVm class
- add maxmem parameter to vm-template-hvm.conf (same as in vm-template.conf)
Without above changes, to test xen-balloon, you must place "maxmem" parameter
(set to some high value - eg half of host memory size - in MB) manually in VM
Xen config file: get config path via qvm-prefs, copy it somewhere, edit and
start VM giving new config via "qvm-start --custom-config=PATH VM-NAME".
Then VM should react to "xl mem-set VM-NAME NEW-MEMORY-SIZE". Currently
increasing from 512 to 1024 is seen by VM as ~20MB additional memory. Perhaps
some kernel parameter missing or sth...