qvm-block doesn't list/expose dom0 loop devices

[nicholas roveda]

I'm using R4.0 rc1.

I wanted to install a Linux distro inside a disk image located in dom0 home, using QEMU in an AppVM.

I've created a new disk image in dom0, set it up (dos partition label and a primary ext4 partition) and attached it with `kpartx` to loopX, but `qvm-block` doesn't list it in the exposed block devices.

So, in order to understand better how Qubes handles this, I've tried to do the same in a VM and see if the dom0 would catch the event, but nothing.
Strangely, when I detached the disk image, both the 'AppVM:loopX - IMG_PATH is available' and 'AppVM:loopX - IMG_PATH is removed' notifications appeared on the screen.

I've read a lot about it, but I think what I found was all related to a < R4.0 version, so I don't know if this issue is related to a intended design or a bug.

Procedure:
[user@dom0 ~]$ dd if=/dev/zero of=/home/user/table.raw bs=2048 count=1
[user@dom0 ~]$ dd if=/dev/zero of=/home/user/root.raw bs=3GB count=1
[user@dom0 ~]$ mkfs.ext4 /home/user/root.raw
[user@dom0 ~]$ cat /home/user/{table,root}.raw > /home/user/root.img
[user@dom0 ~]$ rm /home/user/{table,root}.raw
[user@dom0 ~]$ truncate -s 3GB /home/user/root.img
[user@dom0 ~]$ fdisk /home/user/root.img
o new DOS partition table
n new partition
p primary
1 first
2048 first sector
\n to the end
[user@dom0 ~] fdisk -l /home/user/root.img
Disk /home/user/root.img: 2.8 GiB, 3000000000 bytes, 5859375 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xda159b44

Device Boot Start End Sectors Size Id Type
root.img1 2048 5859374 5857327 2.8G 83 Linux

[user@dom0 ~] sudo kpartx -a /home/user/root.img
[user@dom0 ~] losetup -l
NAME SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE DIO
/dev/loop40 0 0 0 0 /home/user/root.img
0
[user@dom0 ~] ls /dev/mapper
control loop40p1 ...

[Unman]

I'm not surprised at this.
Can you use qvm-block -A to attach a file? Wouldnt that be
easier for you?

[David Hobach]

-A is 3.2 notation and the file support was removed in 4.0rc1 I believe
(it was less reliable than loop devices in 3.2 already anyway).

I don't know what the problem is though: qvm-block l lists everything in
4.0rc1 that is attached to some VM (recently tested exactly that by
chance) - including loop devices.

Just do this in dom0:
losetup --show -f [file]
qvm-block a [VM] dom0:loop[i]
qvm-block l

Maybe you're also running at an older version than me though; I did one
testing-repo-upgrade in dom0 & fedora template VM recently.

[nicholas roveda]

`qvm-block` lists all vms loop device, but no the dom0 ones.

I've checked with `udevadm info -q all` and I've noticed loop devices don't have the 'QUBES_EXPOSED' property.

Can you try to reproduce my situation and show me your `udevadm` output for your loop device?
Thanks a lot.

[David Hobach]

Yes, qvm-block currently doesn't work with dom0 loop devices.

Tracked here: https://github.com/QubesOS/qubes-issues/issues/3084

[Stickstoff]

Hello everyone,

My backupscript broke while upgrading to Qubes4, as my script cycles
through all VMs .img files in dom0, mounts them in a dedicated backup-vm
and backups the contents away from there.

Is there a fix or workaround, to be able to mount (VM) .img files
residing in dom0 to a VM?
Also, I could get a self-created .img file in dom0 to mount with
qvm-block. I fail with the exact same command on qubes' private.img,
although it is assigned a loop device just fine.

dev=$(sudo losetup -f --show /tmp/file.img)
qvm-block attach --ro -o frontend-dev=xvdx backupvm dom0:${dev##*/}
*this works*

dev=$(sudo losetup -f --show /var/lib/qubes/appvms/sys-net/private.img)
qvm-block attach --ro -o frontend-dev=xvdx backupvm dom0:${dev##*/}
--> qvm-block: error: backend vm 'dom0' doesn't expose device 'loop62'

I checked that loop62 is indeed mounted as the expected private.img.

So another option for me would be to somehow manipulate the qubes .img
containers so they are "exposed by dom0" just like selfcreated .img
containers?

And finally: in the git issue, it was advised against "xl block-attach"
and "xl block-detach". Any hints what would happen? Is it inefficient,
or bugged, or does it risk data corruption?


Thank you in advance!

Stickstoff

[awokd]

On Fri, July 27, 2018 8:44 pm, Stickstoff wrote:

>
> Is there a fix or workaround, to be able to mount (VM) .img files
> residing in dom0 to a VM?

Not exactly what you're asking for, but you can make a (large) AppVM, copy
the .img inside it, and mount it there.

[Stickstoff]

I did some more tests.
The difference is, if the VM is in a 'appvms' parent folder, 'qvm-block
attach' fails. When the VM folder is moved elsewhere, with the same
flags and rights, qvm-block works. Even if the appvms folder is just
renamed, qvm-block works. Renaming it back, it fails again.

/test/appvms/test/private.img fails
/test/appvms-1/test/private.img works
/test/appvms/private.img works

It seems like this is hardcoded.
Why is this hardcoded, why shall we not 'qvm-block attach' VM .img files?
If there is a reason to prevent the original VM .img containers to be
'qvm-block attach'ed (maybe because they might be in use already?),
would it be fine to 'qvm-block attach' .img files in other places, even
if they have 'appvms' etc as their parent-parent folder?
(sure, it has security implications to let a VM access other VMs .img
files. I accept this to have faster and more convenient remote backups)
My usecase is to create a local btrfs snapshot, and 'qvm-block attach'
the VM .img files from that snapshot to a backup VM. For example
/snapshots/2018-07-29/appvms/sys-net/private.img.
I could rename the appvms folder in my snapshot, but would prefer a
cleaner solution and to understand the reasoning behind the current
behavior first.

Thank you for the idea.
It would work, but would take a long time, copying, or just rsync'ing
all VMs .img containers away.
If it comes to that route, I would first try with filesystem-level
syncing and backupping.


Thank you for input!

Stickstoff

[Stickstoff]

Hello everybody,

On 07/29/2018 09:33 AM, Stickstoff wrote:
>
> I did some more tests. The difference is, if the VM is in a 'appvms'
> parent folder, 'qvm-block attach' fails. When the VM folder is moved
> elsewhere, with the same flags and rights, qvm-block works. Even if
> the appvms folder is just renamed, qvm-block works. Renaming it
> back, it fails again.
>
> /test/appvms/test/private.img fails
> /test/appvms-1/test/private.img works> /test/appvms/private.img works

Back then, this seemed to work in a shell, but failed with the same
commands in a script.
I checked again now, a month later, and can't get any blockfiles to
attach to a VM via qvm-block, no matter the path. Maybe some Qubes parts
updated in the meantime. Moving a blockfile to /tmp gives a traceback
though, and not the regular "dom0 doesn't expose device".
Relevant github issues, all closed, in [2].

> It seems like this is hardcoded. Why is this hardcoded, why shall we
> not 'qvm-block attach' VM .img files? If there is a reason to
> prevent the original VM .img containers to be 'qvm-block attach'ed
> (maybe because they might be in use already?),

I found hints in [1] about a file
> /var/lib/qubes/.qubes-exclude-block-devices
which seems to "not expose loop files pointing inside". Deleting that
file didn't fix qvm-block, it still fails with the same "not exposed" error.

At this point, I am out of knowledge and ideas.
I'll pay a bounty of 0.1 BTC to anyone who can fix qvm-block to attach
VM image files from dom0 to a VM, the way it worked in Qubes R3.2.
Limited to September 2018 for now. I'll gladly use a 2-of-3 multisig
escrow for freezing the bounty, with a reputable Qubes community member
as third party.

> My usecase is to create a local btrfs snapshot, and 'qvm-block
> attach' the VM .img files from that snapshot to a backup VM. For
> example /snapshots/2018-07-29/appvms/sys-net/private.img. I could
> rename the appvms folder in my snapshot, but would prefer a cleaner
> solution and to understand the reasoning behind the current behavior
> first.

Thanks,

Stickstoff



[1]
https://github.com/marmarek/qubes-core-admin-linux/commit/6ba03ed65bc5aecde7a123bdafbf524a344853f4

[2]
https://github.com/QubesOS/qubes-issues/issues/2126
https://github.com/QubesOS/qubes-issues/issues/3073
https://github.com/QubesOS/qubes-issues/issues/3084
https://github.com/QubesOS/qubes-issues/issues/3172
https://github.com/QubesOS/qubes-issues/issues/3186

[Thomas Papenkort]

I have run into the same problem for backups when switching to qubes 4.0 and
found this workaround:

# a file cannot be attached if it is in directory /var/lib/qubes/appvms, so create a link first
ln /var/lib/qubes/appvms/$1/private.img /home/user/private.img
LOOPDEV=`sudo losetup -f`
sudo losetup $LOOPDEV /home/user/private.img
qvm-block attach -o frontend-dev=xvds -o read-only=true backupvm dom0:$(basename "$LOOPDEV")

[backup happens here]

qvm-block detach backupvm dom0:$(basename "$LOOPDEV")
sudo losetup -d $LOOPDEV
rm /home/user/private.img

[Stickstoff]

Thank you a lot for your help!
I got it to work finally. In fact, it's a combination of the two details:
- get the .img file to another path. It can't stay in
"appvms/VMNAME/private.img". Get a hardlink elsewhere, or rename
"appvms" (and "vm-templates"), both are fine.
- you still have to delete the .qubes-exclude-block-devices file, if you
renamed "appvms" or the path to your hardlink contains this file.

I'm happy to finally have it figured out, so I don't need to compromise
security for backups (regular Linux, or network in DOM0, or booting
something else for backups etc).

Thomas, please drop me a Bitcoin address.

Cheers,

Stickstoff

[David Hobach]

On 09/12/2018 04:51 PM, Stickstoff wrote:
> On 09/11/2018 03:52 PM, Thomas Papenkort wrote:
>> I have run into the same problem for backups when switching to qubes 4.0 and
>> found this workaround:
>>
>> # a file cannot be attached if it is in directory /var/lib/qubes/appvms, so create a link first
>> ln /var/lib/qubes/appvms/$1/private.img /home/user/private.img
>> LOOPDEV=`sudo losetup -f`
>> sudo losetup $LOOPDEV /home/user/private.img
>> qvm-block attach -o frontend-dev=xvds -o read-only=true backupvm dom0:$(basename "$LOOPDEV")
>>
>> [backup happens here]
>>
>> qvm-block detach backupvm dom0:$(basename "$LOOPDEV")
>> sudo losetup -d $LOOPDEV
>> rm /home/user/private.img
>
> Thank you a lot for your help!
> I got it to work finally. In fact, it's a combination of the two details:
> - get the .img file to another path. It can't stay in
> "appvms/VMNAME/private.img". Get a hardlink elsewhere, or rename
> "appvms" (and "vm-templates"), both are fine.
> - you still have to delete the .qubes-exclude-block-devices file, if you
> renamed "appvms" or the path to your hardlink contains this file.

Below in bash what I use since at least 4.0rc1.

Anyway what you're describing is considered a feature, not a bug (I
recall when it was introduced as part of a bug report I had made in the
beginning of 4.0rc1 about qvm-block not supporting files at all). I
think it was a udev rule one-liner checking the path back then.

I'd suggest creating a feature request for a force flag bypassing it and
maybe mention that you'd be willing to donate for that.

I'm not sure whether it's meant to be officially supported though as
they might go away from sparse files with the recent introduction of
qvm-pool etc. (wild guess).

KR
David

----------------------

#createDomZeroLoopDeviceIfNecessary [dom0 path]
#returns: created loop device or previously used one (incl. /dev/); sets
a non-zero exit code, if no device could be created
#create a loop device from the given file path in dom0, if necessary (no
old one does exist)
function createDomZeroLoopDeviceIfNecessary {
local domZeroPath="$1"

#do we have a previously used device?
local oldDev="$(losetup -j "$domZeroPath" | grep -Eo '^/dev/loop[0-9]+')"

if [ -n "$oldDev" ] ; then
echo "$oldDev"
else
#no old device --> create a new one
#we use the exit code as ours
sudo losetup -f --show "$domZeroPath"
fi
}

#getVMDeviceNameForAttached [device in dom0] [VM]
#get the name for the device created in the given VM during execution of
the qvmBlockAttachFromDomZeroTo function
#returns: name of the attached device in the given VM (without the /dev/
prefix) or an empty String, if no such device exists (not attached anymore)
function getVMDeviceNameForAttached {
local dev="$1"
local vm="$2"
local regex="^dom0:$dev\s+.*\s+${vm}\s+\(.*frontend-dev=([a-z0-9]+).*\).*$"

#run the regex against qvm-block & return the result
qvm-block l | sed -r -n "s/$regex/\\1/p"
}

#qvmBlockAttachFromDomZeroTo [result] [source path in dom0 (must be a
file)] [target VM to attach the source path to] [optional: ro flag]
#[ro flag]: optional flag, that - if set to 1 - makes sure the file is
attached as read-only to the target VM
#returns: device created in the VM as the result variable or an empty
result variable in the case of errors
function qvmBlockAttachFromDomZeroTo {
local result="$1"
local path="$2"
local targetVM="$3"
local empty=""
local dev=""
local rwOption=""
[ $4 -eq 1 ] && rwOption="-o read-only=yes"

#default = error
eval $result="'$empty'"

#we need to create a pseudo file as Qubes 4.0rc1 will attempt to prevent
qvm-block usage for its private.img files
#hard links work to bypass that, but need to be on the same drive (/tmp
doesn't work)
local pfile="$DOM0_TEMP_DEV_PATH/$(echo "$path" | md5sum | cut -d " " -f1)"
ln "$file" "$pfile"

#unfortunately qvm-block l has issues for files, so we create a loop
device first and use that one
#also see: R3.0: qvm-block doesn't work well on files
(https://groups.google.com/forum/#!msg/qubes-users/IotETu-gsm4/FO2GOu5pBwAJ)
#for 4.0rc1 I'm not even sure whether it supports files...
dev="$(createDomZeroLoopDeviceIfNecessary "$pfile")"
[ $? -ne 0 ] && return
dev="${dev/\/dev\//}"

#run qvm-block
qvm-block a $rwOption "$targetVM" "dom0:${dev}"
[ $? -ne 0 ] && return

#get the return value and update the return var
lastAttached="$(getVMDeviceNameForAttached "$dev" "$targetVM")"
eval $result="'$lastAttached'"
}