Bug#1061725: libvirt-daemon: Deleting external snapshot for non-running system VM fails with Permission Denied

[Martin Pitt]

Package: libvirt-daemon
Version: 10.0.0-1

When creating a trivial VM and doing an external snapshot if the VM is *not* running,
deleting the snapshot fails. As root:

qemu-img create -f qcow2 /var/lib/libvirt/images/test1.qcow2 10G
virt-install --memory 50 --pxe --virt-type qemu --os-variant alpinelinux3.8 --wait 0 --name test1 --disk target=vda,path=/var/lib/libvirt/images/test1.qcow2
virsh destroy test1
virsh snapshot-create-as --domain test1 --name snap1 --disk-only --diskspec vda,snapshot=external,file=/var/lib/libvirt/images/test1-snap1
virsh snapshot-delete test1 snap1

The last command fails with

| error: Failed to delete snapshot snap1
| error: internal error: unable to execute QEMU command 'block-commit': Could not open '/var/lib/libvirt/images/test1.qcow2': Permission denied

The error message may be a bit misleading -- this is the *image* file, which
has wide-open libvirt-qemu:libvirt-qemu 666 permissions. "test1-snap1" in the
same directory is much more restrictive root:root 644; but even trying to
chown/chmod that doesn't unbreak this. So perhaps it's trying to do something
funky to the actual image file after all.

It also happens with the automatically created disk image, which will then be
IDE "hda", not virtio "vda":

virt-install --memory 50 --pxe --virt-type qemu --os-variant alpinelinux3.8 --wait 0 --name test2
virsh destroy test2
virsh snapshot-create-as --domain test2 --name snap1 --disk-only --diskspec hda,snapshot=external,file=/var/lib/libvirt/images/test2-snap1
virsh snapshot-delete test2 snap1

It works when doing a snapshot from a *running* VM, either disk-only or with memory:

virsh start test1
virsh snapshot-create-as --domain test1 --name snap2 --disk-only --diskspec vda,snapshot=external,file=/var/lib/libvirt/images/test1-snap2
virsh snapshot-create-as --domain test1 --name snap3 --memspec file=/var/lib/libvirt/qemu/snapshot/test1-snap3-memory --diskspec vda,snapshot=external,file=/var/lib/libvirt/images/test1-snap3

Then both snap2 and snap3 can be deleted. But still not snap1, so the running
state matters at the time of snapshot creation, not deletion.

This also happens with libvirt 9.0.0-4 in Debian stable and libvirt
8.0.0-1ubuntu7.8 in Ubuntu 22.04 LTS, but curiously not with libvirt
9.6.0-1ubuntu1 in Ubuntu 23.10. It also works fine in Fedora, CentOS/RHEL 8/9,
and Arch Linux, so this is somehow specific to Debian.

I tried `aa-teardown` just in case it's apparmor, but that doesn't seem to
influence it.

There is no useful/relevant journal message about this other than the
"Permission denied" line that's already on stderr.

I also tried this as user:

virt-install --memory 50 --pxe --virt-type qemu --os-variant alpinelinux3.8 --wait 0 --name test1
virsh destroy test1
virsh snapshot-create-as --domain test1 --name snap1 --disk-only --diskspec hda,snapshot=external,file=$HOME/.local/share/libvirt/images/test1-snap1
virsh snapshot-delete test1 snap1

This works as well. So this is specific to qemu:///system, session works fine.

Martin

[Martin Pitt]

Control: retitle -1 libvirt-daemon: Deleting external snapshot for non-running system VM fails with AppArmor

when stracing libvirt, this is what happens:

6557 openat(AT_FDCWD, "/var/lib/libvirt/images/test2.qcow2", O_RDWR|O_CLOEXEC) = -1 EACCES (Permission denied)
6557 sendmsg(13, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="{\"id\": \"libvirt-443\", \"error\": {\"class\": \"GenericError\", \"desc\
": \"Could not open '/var/lib/libvirt/images/test2.qcow2': Permission denied\"}}\r\n", iov_len=142}], msg_iovlen=1, msg_controllen=0, msg_flags
=0}, 0 <unfinished ...>

and the most recent geteuid() call responded with "0". So it actually *does*
smell like an AppArmor issue, even though it's weird that it would work for a
running VM then. Running `aa-teardown` before the creation of the VM doesn't
work, nor does "aa-complain libvirtd". But after `dpkg -P apparmor; reboot` it
does work.

So AppArmor breaks this without even logging about it, i.e. some "deny" rule. I
don't know how to make AA log deny rules -- the profile has tons of them
(albeit to /proc, /dev/, etc.), and it's further complicated by the dynamic
profile creation through virt-aa-helper.

As this works in current Ubuntu, it's perhaps worth looking at
https://patches.ubuntu.com/libv/libvirt/libvirt_9.6.0-1ubuntu2.patch
The most plausible one may be debian/patches/ubuntu-aa/0031-virt-aa-helper-Ask-for-no-deny-rule-for-readonly-dis.patch
but that requires rebuilding libvirt. But also, that patch is from 2017, and
it's still broken in Ubuntu 22.04.

[Martin Pitt]

I can't make head or tail of this. aa-complain still enforces deny
rules, there is no (discoverable) way to log deny rules, and

grep -r deny /etc/apparmor.d | grep virt | grep -v /sys | grep -v /dev

doesn't show anything which would apply to /var/lib/libvirt/.

`aa-disable /etc/apparmor.d/libvirt/libvirt-a5aa3a67-6967-43a5-9d61-b0b380bd14e6` also
doesn't work because it references a non-existing
libvirt/libvirt-a5aa3a67-6967-43a5-9d61-b0b380bd14e6.files.

The only thing that works is

aa-disable libvirtd
systemctl restart libvirtd

(That requires apparmor-utils)

After that, the snapshot-delete command works.

I don't know what else I could try here to debug this properly, so a hint from
someone AppArmor-savvy would be much appreciated.