Creating non-qubes vms

[Abel Luck]

Hi,

Is it possible to create non-qubes VMs from dom0?

Part of my work involves testing in other vms (livecd isos, special
linux distros, etc).

I could install kvm/qemu in an AppVM, I suppose, but that would add a
huge performance overhead.

The simple case is I have an image file in one AppVM and I want to boot
a VM from that image file (either with the image mounted as a CD or as
the hard disk).

I don't need window manager support, nor clipboard support, but I would
need network access. A single window with the booted OS inside would
suffice (rather than app-level window integration).

Is this possible?

~abel

[Marek Marczykowski]

This should be possible - create standard Xen VM. Note that this setup will be
not as secure as normal Qubes VM - device model (qemu) will be running in
dom0, which introduce _huge attack factor_ (most of recent Xen Security
Advisories was related to such setup).

The only non-standard thing is getting memory from qubes memory manager.

Assuming you need 1024MB RAM for VM:
$ python
>>> from qubes.qubes import QMemmanClient
>>> qc = QMemmanClient()
>>> qc.request_memory (1024*1024*1024)

(do not close python, start VM now from other console)

>>> qc.close()

VM reboot will not work - you need shutdown VM and then repeat above procedure.

Someone already have done this - you can search the list archive.

You can also find useful information on wiki.xen.org.

I have some early notes from Windows support for Qubes (not tested if still
works):

Commands in dom0:
mkdir /var/lib/qubes/hvm
truncate -s 20G /var/lib/qubes/hvm/win7.img
sudo qvm-dom0-update tigervnc

VM startup:
xl create win7-hvm.cfg
vncviewer localhost

Example Xen config attached.

Network setup options:
1. Using PV drivers (which is the case in most Linux guest systems):
copy "vif" line from some VM config (/var/lib/qubes/appvms/*/*.conf) and
change IP (and MAC) to some unused address, example:
vif = [
'mac=00:16:3E:5E:6C:FF,script=/etc/xen/scripts/vif-route-qubes,ip=10.137.2.127,backend=firewallvm'
]

2. Using qemu-dm - via dom0 (will work without PV drivers - eg Windows):
leave vif line as in attached example
commands in dom0 (as root) before VM startup (10.138.0.3 is VM IP):
sysctl net.ipv4.ip_forward=1
qubes-dom0-network-via-netvm up
iptables -t nat -A POSTROUTING -s 10.138.0.3 -j MASQUERADE
iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -s 10.138.0.3 -j ACCEPT

In both cases you need to setup IP inside of VM manually. Default route
directly via eth0 (ip route add default dev eth0). Copy DNS settings from some
other VM (something like 10.137.2.1 and 10.137.2.254).

The second option (network via dom0) introduce even bigger attack factor, as
dom0 will be exposed to network attacks from your HVM and netvm.

Please let me know if it works for you, maybe I will place above instruction
on our wiki.

--
Best Regards / Pozdrawiam,
Marek Marczykowski
Invisible Things Lab

[abb]

These instructions work for me, but even for PV drivers one needs to fix networking rules in netvm with something like
---
fwcfg() {
    vmip=$1

    iptables -A FORWARD -s $vmip -p udp -d 10.137.1.1   --dport 53 -j ACCEPT
    iptables -A FORWARD -s $vmip -p udp -d 10.137.1.254 --dport 53 -j ACCEPT
    iptables -A FORWARD -s $vmip -p icmp -j ACCEPT
    iptables -A FORWARD -s $vmip -p tcp -d 10.137.255.254 --dport 8082 -j DROP
    iptables -A FORWARD -s $vmip -j ACCEPT
}

fwcfg 10.137.2.127
fwcfg 10.137.2.128
fwcfg 10.137.2.129
---

Disk performance is awful. On SSD it is 2MB/s, on mechanic disk it is some 0.6MB/s and I can hear that the disk is being trashed really badly.

I've managed to install Win7 (no networking) and NetBSD 6.0. Tried also BT5R3 but this one trashes the disk particularly badly, never made it through to the end of the install. NetBSD seems to run happily, but network is not working, says "re0: watchdog timeout".

Same results for kernel 3.2 and 3.4.

Have not tried 'qvm-create --hvm' yet.

--
Best Regards / Pozdrawiam,
Marek Marczykowski
Invisible Things Lab

name = "win7"

builder='hvm'
memory=1024
#viridian=1
kernel='hvmloader'
#acpi=1
#apic=1
boot='dca'
device_model='qemu-dm'
#pae=1
usbdevice='tablet'
sdl=0
vnc=1
vnclisten='127.0.0.1'
disk=[ 'file:/var/lib/qubes/hvm/win7.img,hda,w' ]
#, 'file:/tmp/win7-install.iso,hdc:cdrom,r' ]
vif=[ 'script=/etc/xen/scripts/vif-nat,ip=10.138.0.4,mac=00:11:22:33:44:55' ]
#tsc_mode = 2
#xen_extended_power_mgmt=0

vcpus = 1

on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'preserve'

[Marek Marczykowski]

On 24.10.2012 23:07, abb wrote:

(...)

> These instructions work for me, but even for PV drivers one needs to fix
> networking rules in netvm with something like
> ---
> fwcfg() {
> vmip=$1
>
> iptables -A FORWARD -s $vmip -p udp -d 10.137.1.1 --dport 53 -j ACCEPT
> iptables -A FORWARD -s $vmip -p udp -d 10.137.1.254 --dport 53 -j ACCEPT
> iptables -A FORWARD -s $vmip -p icmp -j ACCEPT
> iptables -A FORWARD -s $vmip -p tcp -d 10.137.255.254 --dport 8082 -j
> DROP
> iptables -A FORWARD -s $vmip -j ACCEPT
> }
>
> fwcfg 10.137.2.127
> fwcfg 10.137.2.128
> fwcfg 10.137.2.129
> ---

Indeed, VMs not managed by Qubes tools don't get firewall rules...

> Disk performance is awful. On SSD it is 2MB/s, on mechanic disk it is some
> 0.6MB/s and I can hear that the disk is being trashed really badly.

This is much better with Qubes-managed HVMs, which uses stubdom to emulate
devices.

> I've managed to install Win7 (no networking) and NetBSD 6.0. Tried also
> BT5R3 but this one trashes the disk particularly badly, never made it
> through to the end of the install. NetBSD seems to run happily, but network
> is not working, says "re0: watchdog timeout".

Does NetBSD have network PV drivers? If so, perhaps have hardcoded dom0 as
backend domain (which is sadly quite popular bug...). You can check 'xl dmesg'
in dom0, if I guessed correctly, there will be some messages about gnttab or so.

> Same results for kernel 3.2 and 3.4.

Where? In dom0? Or netvm/firewallvm?

[Radosław Szkodziński]

W dniu 24.10.2012 23:07, abb pisze:

> Disk performance is awful. On SSD it is 2MB/s, on mechanic disk it is some
> 0.6MB/s and I can hear that the disk is being trashed really badly.

Install GPLPV drivers and it gets reasonable. (Good enough here on an
SSD to get 8 stars in Win 7 system benchmark.)

--
Radosław Szkodziński

[Marek Marczykowski]

Note that official GPLPV drivers don't work with network backend != dom0 (eg
netvm). You can either compile and install drivers from our repo
(win-pvdrivers), or use official one, but do not install network driver.

[abb]

According to their release info supposed to have it. And it detected the interface re0.
 

If so, perhaps have hardcoded dom0 as
backend domain (which is sadly quite popular bug...). You can check 'xl dmesg'
in dom0, if I guessed correctly, there will be some messages about gnttab or so.

Ok, will check.
 

> Same results for kernel 3.2 and 3.4.

Where? In dom0? Or netvm/firewallvm?

Initial tests were done for 3.2 in both dom0 and vms.
Then I've installed 3.4 to test pvusb stuf. Certainly in dom0, I will check if I upgraded only the development VM or the template, probably the template.

[abb]

Here is a relevant installation guide, I have not tried it yet:
http://wiki.xen.org/wiki/How_to_install_a_NetBSD_PV_domU_on_a_Debian_Squeeze_host_%28Xen_4.0.1%29
 
I did install from ISO and probably ended up without PV drivers, see attached screenshot.

Ok, will check.
 

> Same results for kernel 3.2 and 3.4.

Where? In dom0? Or netvm/firewallvm?

Initial tests were done for 3.2 in both dom0 and vms.
Then I've installed 3.4 to test pvusb stuf. Certainly in dom0, I will check if I upgraded only the development VM or the template, probably the template.

3.4 was in both dom0 and vms.

[Marek Marczykowski]

Ok, looks to be emulated hardware, not PV. Anyway this still can be the
problem with unsupported backend != dom0. Don't remember details, but in case
of "non-qubes" HVM (without PV drivers) I've managed to get network in VM only
by routing through dom0...

Above instruction looks promising, I suggest you create standalone VM (based
on any template, the Linux inside will be overridden anyway), then place the
netbsd kernel in "kernels" subdir of VM (as "vmlinuz") and some placeholder
"initramfs". Then set kernel (via qvm-prefs) to "none".

During startup of such VM hit Ctrl-C while waiting for qrexec agent, then you
can access console via "sudo xl console <vmname>".

If you want to stick with HVM, you can use qvm-create --hvm, with new xen etc.
It really have many things fixed (eg non-dom0 backend support).

[abb]

These instructions worked for me.

Create standalone VM with default template, replaced vmlinuz with netbsd-INSTALL_XEN3_DOMU kernel. During setup chosen to install on xbd1 hard disk. Could not figure out how to attach CD, 'qvm-start --cdrom=...' says VM does not support attaching disks. DHCP client in NetBSD didn't work during installation, so I had to manually configure the networking. The installation went fine, enabled SSHD during post-install configuration. Replaced the kernel with netbsd-XEN3_DOMU as the wiki says. It fails to boot automatically, need to explicitly specify xbd1a as root device when prompted. After bootstrap all works just fine, networking seems to work too (xennet0).

If you want to stick with HVM, you can use qvm-create --hvm, with new xen etc.
It really have many things fixed (eg non-dom0 backend support).

I tried 'qvm-create --hvm', got Windows7 installed just fine, networking works out of the box. Didn't try custom PV drivers yet.

[abb]

Are there plans to use pygrub?
 

[Juergen Schinker]

Due to this email I have now a new shiny NetBSD box ........ well Thank you

---

--
 
 

[Joanna Rutkowska]

On 10/27/12 04:29, abb wrote:
>> Above instruction looks promising, I suggest you create standalone VM
>> > (based
>> > on any template, the Linux inside will be overridden anyway), then place
>> > the
>> > netbsd kernel in "kernels" subdir of VM (as "vmlinuz")
>
> Are there plans to use pygrub?
>
>

PyGrub is inherently insecure, because it does processing of
AppVM-supplied data (e.g. vmlinuz) on the Dom0. I recall there were some
bugs allowing to 0wn Dom0 from by exploiting bugs in code used for
vmlinuz decompression or something.

We might, however, consider using PVGrub, which does everything in the VM.

BTW, the reason we decided to separate AppVM's kernels/initramfses from
the AppVM's root filesystem was to support our (still not implemented)
untrusted storage domain (See chapter 7 in Qubes Arch Spec PDF).

If we didn't keep at least the AppVM's kernel in the trusted place
(Dom0) then the AppVM would have no chances to verify whether its
root.img (as served by the untrusted storage domain) was indeed
correctly signed or not. So, at least the trusted kernel is needed for
this. This doesn't mean that Dom0 was taking part in "being a storage
backend", as the kernel/initramfs were only used at domain build time,
and subsequently, once the VM fired up, there was no more interaction
with Dom0 (e.g. to retrieve blocks of kernel/initramfs).

We've been recently discussing with Marek whether we want to invest
resources into building Untrusted Storage support into Qubes 2, and
we're currently aiming toward not doing this, because this would be
require lots of effort, and at the same time the gain is not
significant: the elimination of the block backend from Dom0 (especially
that we plan to have default USB domain Qubes 2, as soon as the kernel
pvusb backend gets mature enough).

Cheers,
joanna.

[abb]

Can't figure out how to fix the root device, probably need to pass something to BSD kernel somehow? I've moved root filesystem to the first disk, but it changed nothing. Alternative solution is to rebuild NetBSD kernel. It worked for me, notes attached in case someone is interested (note it uses the first harddisk there).

[Marek Marczykowski]

You can try some qvm-prefs <vmname> -s kernelopts.

[Radosław Szkodziński]

W dniu 27.10.2012 12:43, Joanna Rutkowska pisze:

> We might, however, consider using PVGrub, which does everything in the VM.
>
> BTW, the reason we decided to separate AppVM's kernels/initramfses from
> the AppVM's root filesystem was to support our (still not implemented)
> untrusted storage domain (See chapter 7 in Qubes Arch Spec PDF).
>
> If we didn't keep at least the AppVM's kernel in the trusted place
> (Dom0) then the AppVM would have no chances to verify whether its
> root.img (as served by the untrusted storage domain) was indeed
> correctly signed or not. So, at least the trusted kernel is needed for
> this.

However, you could use PVGrub as the platform for this. Kernels can then
remain inside. However, PVGrub would need to be extended to communicate
with Xen "TPM", much like AntiEvilMaid.

> This doesn't mean that Dom0 was taking part in "being a storage
> backend", as the kernel/initramfs were only used at domain build time,
> and subsequently, once the VM fired up, there was no more interaction
> with Dom0 (e.g. to retrieve blocks of kernel/initramfs).

--
Radosław Szkodziński

[Fernando Parodi]

Hello, I would like to create a hvm vm with new functionality -hvm (this already do it in qubes-manager), I wanted to know what would be the steps to install win 7, saw in qubes-builder, windows-build, win-iso -iso.spec dirs and win, win-mkinstalliso.sh, win-mksrcimg.sh files (i run the script but some is missing), could please tell me the steps, from what I read in previous post will take a closed version of drivers? How i  can get this drivers ?
O please guided-me for what documentation i need to read.

Best regards,

PD: Sorry for my english

[Marek Marczykowski]

Here is the answer for your question:
https://groups.google.com/group/qubes-devel/browse_thread/thread/f6743a1d679a62fe

You can use the same way to start VM with win7-installer.iso attached
(qvm-start --cdrom=path/to/win7.iso).

[pr...@arcor.de]

On Saturday, October 27, 2012 at 4:14:39 AM UTC+2, abb wrote:

These instructions worked for me.

Create standalone VM with default template, replaced vmlinuz with netbsd-INSTALL_XEN3_DOMU kernel. During setup chosen to install on xbd1 hard disk. Could not figure out how to attach CD, 'qvm-start --cdrom=...' says VM does not support attaching disks. DHCP client in NetBSD didn't work during installation, so I had to manually configure the networking. The installation went fine, enabled SSHD during post-install configuration. Replaced the kernel with netbsd-XEN3_DOMU as the wiki says. It fails to boot automatically, need to explicitly specify xbd1a as root device when prompted. After bootstrap all works just fine, networking seems to work too (xennet0).

I tried to follow the instructions above with NetBSD_7.0.2 but I am stuck in  pointing to the base directory. Like described above I wasn't able to attach a DVD/CD drive either, so I chose "local directory". I type in the path to the binaries needed for the installation but I always get an error "path does not exist". Can someone help me with this. I am sensing some trivial mistake but can't figure it out yet. (I copied the contents of the latest ISO into a personal folder in dom0, to which I am pointing: home/user/Downloads/netbsd

Thanks for taking the time to read!

[Unman]

These are pretty much the same instructions as at
www.qubes-os.org/doc/netbsd and worked for me. That was some time ago.

What error are you getting when trying to atatch the cdrom drive?

[Антон Чехов]

On Tuesday, January 10, 2017 at 2:49:34 PM UTC+1, Unman wrote:

These are pretty much the same instructions as at
www.qubes-os.org/doc/netbsd and worked for me. That was some time ago.

What error are you getting when trying to atatch the cdrom drive?

When trying to start the VM with "qvm-start --cdrom=..." I get "VM does not support attaching disks". So I just start the VM and access the console during startup. When trying to attach the cdrom I get:

"Command failed
 /sbin/mount -rt cd9660 /dev/sr0  /mnt2"

and a path to a local directory does not work either.

[Unman]

Ok, well you're trying to create an appVM which wont support the --cdrom
options at start. (You can use that option with HVM)

I dont know how you are trying to attach the disk image: in my
experience, qvm-block and native xl commands don't work, but perhaps I
am missing something.

As a workaround, I simply replaced private.img with the NetBSD iso.
Boot the qube, interrupt and attach to console.
Install to xbd0; then the CD is available at xbd1a
You can always reinstate private.img if you want to use that disk
space also.

See if that works for you

unman

[Unman]

It occurs to me that if you replace root.img with the NetBSD iso then
the qube would boot straight in to installation mode. Havent tried it
but it seems likely.
Obviously you may want to remove the installer afterwards, but as you
will be specifying the disk to boot from it probably wont be necessary.

[Антон Чехов]

On Wednesday, January 11, 2017 at 12:43:06 AM UTC+1, Unman wrote:

 
It occurs to me that if you replace root.img with the NetBSD iso then
the qube would boot straight in to installation mode. Havent tried it
but it seems likely.
Obviously you may want to remove the installer afterwards, but as you
will be specifying the disk to boot from it probably wont be necessary.

Thanks a lot for your effort to help! It is much appreciated.
Unfortunately nothing changed for me... I can't get to load the ISO/CD (or path to a folder).
Like you say, It works as HVM but I can't get it to work as a Standalone VM like it is explained in the docs. I am not familiar with netbsd and I don't have much knowledge about Xen either. That is certainly part of the problem. I will try again, when I have the time. In the meantime I will read more about how this should work theoretically.

[Unman]

Just for the record here's what to do:
Create a standalone qube with default template - we'll call it netbsd.
Download the NetBSD installer iso, and XEN3_DOMU.gz files and transfer
them to Dom0. Copy them to /var/lib/qubes/appvms/netbsd

In Dom0:
qvm-prefs netbsd -s kernel none
mv netbsd-INSTALL_XEN3_DOMU.gz kernels/vmlinuz
mv root.img root.img.bak
mv netBSD-7.0.2-amd64 root.img

qvm-start netbsd
CTRL+C at Waiting for qrexec daemon
'sudo xl console netbsd' will get you console with installation loaded.

Work through the installation, installing to xbd1.
When prompted to install software, select "From CD" - change the device
name to xbd0a

When installation is complete and you have configured as you wish,
shutdown the qube.

Back in Dom0:
mv root.img.bak root.img
mv netBSD-XEN3_DOMU,gz kernels/vmlinuz

Now your netbsd qube is ready to run. Start and interrup as before -
attach console and you will see boot messages.
When prompted set root device to xbd1a - NetBSD will boot normally.

It wont take long. Try it when you have time.

unman