Bug#1061634: mmdebstrap: possible mmdebstrap-autopkgtest-build-qemu improvements

[Francesco Poli (wintermute)]

Package: mmdebstrap
Version: 1.4.1-1
Severity: normal

Hello Johannes!

I've been able to create a QEMU/KVM virtual machine image for autopkgtests:

$ mkdir -p ~/var/cache/sbuild/
$ cd /dev/shm
$ TMPDIR=/dev/shm mmdebstrap-autopkgtest-build-qemu \
--size=25G --boot=efi sid sid-amd64.img
$ chmod 660 sid-amd64.img
$ mv -i sid-amd64.img ~/var/cache/sbuild/

It is able to run autopkgtests:

$ autopkgtest --output-dir ./${SRCPKG}_autopkgtest.out \
--summary ./${SRCPKG}_autopkgtest.summary \
--apt-upgrade -B ./${SRCPKG}_amd64.changes -- \
qemu --boot=efi --overlay-dir /dev/shm \
~/var/cache/sbuild/sid-amd64.img

My comments / suggestions for improvements:

* I had to manually fix the permissions, since the image file was
originally created with the following weird ones:

$ ls -l --si sid-amd64.img
-rw-r-xrwx 1 $USER $USER 27G Jan 27 17:48 sid-amd64.img

I think the file permissions should be set (possibly at the end of the
mmdebstrap-autopkgtest-build-qemu run) as they would "naturally"
result from the user's umask:

$ cd /dev/shm
$ touch foo
$ ls -l --si foo
-rw-rw---- 1 $USER $USER 0 Jan 27 19:00 foo

That's why I manually changed the permissions at the end:

$ chmod 660 sid-amd64.img
$ ls -l --si sid_amd64.img
-rw-rw---- 1 $USER $USER 27G Jan 27 17:48 sid-amd64.img

I suggest that mmdebstrap-autopkgtest-build-qemu applies this fix
automatically.

* I was surprised to see a 25 GiB image fit into a 7.7 GiB filesystem:

$ cd /dev/shm
$ ls -l -h sid-amd64.img
-rw-rw---- 1 $USER $USER 26G Jan 27 17:48 sid-amd64.img
$ df -h .
Filesystem Size Used Avail Use% Mounted on
tmpfs 7.7G 765M 7.0G 10% /dev/shm

Well, the mystery is solved by looking at the allocated size:

$ ls -l -h -s sid-amd64.img
662M -rw-rw---- 1 $USER $USER 26G Jan 27 17:48 sid-amd64.img

Would it be less confusing, if mmdebstrap-autopkgtest-build-qemu created
.qcow2 images?


-- System Information:
Debian Release: trixie/sid
APT prefers testing
APT policy: (800, 'testing'), (500, 'unstable')
Architecture: amd64 (x86_64)

Kernel: Linux 6.5.0-5-amd64 (SMP w/4 CPU threads; PREEMPT)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages mmdebstrap depends on:
ii apt 2.7.10
ii perl 5.38.2-3
ii python3 3.11.6-1

Versions of packages mmdebstrap recommends:
ii arch-test 0.21-1
ii fakechroot 2.20.1+ds-15
ii fakeroot 1.33-1
ii gpg 2.2.40-1.1+b1
ii libdistro-info-perl 1.7
ii libdpkg-perl 1.22.2
ii mount 2.39.3-6
ii uidmap 1:4.13+dfsg1-3+b1

Versions of packages mmdebstrap suggests:
pn apt-transport-tor <none>
ii apt-utils 2.7.10
ii ca-certificates 20230311
ii debootstrap 1.0.134
ii distro-info-data 0.60
ii dpkg-dev 1.22.2
pn genext2fs <none>
ii perl-doc 5.38.2-3
pn qemu-user <none>
pn qemu-user-static <none>
pn squashfs-tools-ng <none>
ii systemd 255.2-4

-- no debconf information

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Francesco Poli (wintermute) (2024-01-27 19:20:08)

> I've been able to create a QEMU/KVM virtual machine image for autopkgtests:
>
> $ mkdir -p ~/var/cache/sbuild/
> $ cd /dev/shm
> $ TMPDIR=/dev/shm mmdebstrap-autopkgtest-build-qemu \
> --size=25G --boot=efi sid sid-amd64.img
> $ chmod 660 sid-amd64.img
> $ mv -i sid-amd64.img ~/var/cache/sbuild/
>
> It is able to run autopkgtests:
>
> $ autopkgtest --output-dir ./${SRCPKG}_autopkgtest.out \
> --summary ./${SRCPKG}_autopkgtest.summary \
> --apt-upgrade -B ./${SRCPKG}_amd64.changes -- \
> qemu --boot=efi --overlay-dir /dev/shm \
> ~/var/cache/sbuild/sid-amd64.img
>
> My comments / suggestions for improvements:
>
> * I had to manually fix the permissions, since the image file was
> originally created with the following weird ones:
>
> $ ls -l --si sid-amd64.img
> -rw-r-xrwx 1 $USER $USER 27G Jan 27 17:48 sid-amd64.img

those are some funny permission bits. When I run
mmdebstrap-autopkgtest-build-qemu I get:

$ ls -lha disk.img
-rw-r--r-- 1 josch josch 26G Jan 28 00:03 disk.img

Looks fine to me.

> I think the file permissions should be set (possibly at the end of the
> mmdebstrap-autopkgtest-build-qemu run) as they would "naturally" result
> from the user's umask:
>
> $ cd /dev/shm
> $ touch foo
> $ ls -l --si foo
> -rw-rw---- 1 $USER $USER 0 Jan 27 19:00 foo

They are set, taking the user's umask into account. What is your umask?

> That's why I manually changed the permissions at the end:
>
> $ chmod 660 sid-amd64.img
> $ ls -l --si sid_amd64.img
> -rw-rw---- 1 $USER $USER 27G Jan 27 17:48 sid-amd64.img
>
> I suggest that mmdebstrap-autopkgtest-build-qemu applies this fix
> automatically.

Lets first understand the problem before adding a workaround.
mmdebstrap-autopkgtest-build-qemu runs this command to set the permissions:

chmod "$(printf %o "$(( 0666 - 0$(umask) ))")" "$IMAGE"

Could you add some debugging output to the script to figure out what went wrong
and where?

> * I was surprised to see a 25 GiB image fit into a 7.7 GiB filesystem:
>
> $ cd /dev/shm
> $ ls -l -h sid-amd64.img
> -rw-rw---- 1 $USER $USER 26G Jan 27 17:48 sid-amd64.img
> $ df -h .
> Filesystem Size Used Avail Use% Mounted on
> tmpfs 7.7G 765M 7.0G 10% /dev/shm
>
> Well, the mystery is solved by looking at the allocated size:
>
> $ ls -l -h -s sid-amd64.img
> 662M -rw-rw---- 1 $USER $USER 26G Jan 27 17:48 sid-amd64.img
>
> Would it be less confusing, if mmdebstrap-autopkgtest-build-qemu created
> .qcow2 images?

I don't understand why sparse files are confusing to you. Why would qcow images
be less confusing? Can you list some reasons why qcow2 should be preferred over
raw images? Compression comes to mind but that is at the cost of doing that
compression in the first place and it is not obvious to me whether cpu cycles
or disk space are more scarce. For example, my platform is an ARM Cortex A53.
To give you some idea how slow it is: it takes 30 seconds before a youtube
video starts playing. At the same time, the system has a 2 TB hard disk. I much
prefer a bigger disk image which needs less CPU when I want to create and use
it. What is your use-case?

Thanks!

cheers, josch

[Francesco Poli]

On Sun, 28 Jan 2024 00:33:06 +0100 Johannes Schauer Marin Rodrigues wrote:

[...]

> Quoting Francesco Poli (wintermute) (2024-01-27 19:20:08)

[...]

> > My comments / suggestions for improvements:
> >
> > * I had to manually fix the permissions, since the image file was
> > originally created with the following weird ones:
> >
> > $ ls -l --si sid-amd64.img
> > -rw-r-xrwx 1 $USER $USER 27G Jan 27 17:48 sid-amd64.img
>
> those are some funny permission bits. When I run
> mmdebstrap-autopkgtest-build-qemu I get:
>
> $ ls -lha disk.img
> -rw-r--r-- 1 josch josch 26G Jan 28 00:03 disk.img
>
> Looks fine to me.

What's *your* umask? Is it Debian default (022), by chance?

>
> > I think the file permissions should be set (possibly at the end of the
> > mmdebstrap-autopkgtest-build-qemu run) as they would "naturally" result
> > from the user's umask:
> >
> > $ cd /dev/shm
> > $ touch foo
> > $ ls -l --si foo
> > -rw-rw---- 1 $USER $USER 0 Jan 27 19:00 foo
>
> They are set, taking the user's umask into account. What is your umask?

Mine is James Bond's umask: 007 ;-)

$ umask
0007

>
> > That's why I manually changed the permissions at the end:
> >
> > $ chmod 660 sid-amd64.img
> > $ ls -l --si sid_amd64.img
> > -rw-rw---- 1 $USER $USER 27G Jan 27 17:48 sid-amd64.img
> >
> > I suggest that mmdebstrap-autopkgtest-build-qemu applies this fix
> > automatically.
>
> Lets first understand the problem before adding a workaround.
> mmdebstrap-autopkgtest-build-qemu runs this command to set the permissions:
>
> chmod "$(printf %o "$(( 0666 - 0$(umask) ))")" "$IMAGE"

This seems to work with Debian default umask (022):

$ printf '%o\n' "$(( 0666 - 00022 ))"
644

but fails whenever a umask includes an octal digit equal to 7, due to
the carryover:

$ printf '%o\n' "$(( 0666 - 00007 ))"
657

Shouldn't this use the bitwise AND combined with the bitwise NOT?

$ umask
0007
$ printf '%o\n' "$(( 0666 & ~0$(umask) ))"
660
$ umask 022
$ umask
0022
$ printf '%o\n' "$(( 0666 & ~0$(umask) ))"
644

I would think that mmdebstrap-autopkgtest-build-qemu should run:

chmod "$(printf %o "$(( 0666 & ~0$(umask) ))")" "$IMAGE"

Does it make sense to you?

>
> Could you add some debugging output to the script to figure out what went wrong
> and where?

Feel free to suggest any further debug that could be useful, in case
the above reasoning is not enough to shed some light on what happened.

>
> > * I was surprised to see a 25 GiB image fit into a 7.7 GiB filesystem:
> >
> > $ cd /dev/shm
> > $ ls -l -h sid-amd64.img
> > -rw-rw---- 1 $USER $USER 26G Jan 27 17:48 sid-amd64.img
> > $ df -h .
> > Filesystem Size Used Avail Use% Mounted on
> > tmpfs 7.7G 765M 7.0G 10% /dev/shm
> >
> > Well, the mystery is solved by looking at the allocated size:
> >
> > $ ls -l -h -s sid-amd64.img
> > 662M -rw-rw---- 1 $USER $USER 26G Jan 27 17:48 sid-amd64.img
> >
> > Would it be less confusing, if mmdebstrap-autopkgtest-build-qemu created
> > .qcow2 images?
>
> I don't understand why sparse files are confusing to you.

I am clearly not very used to seeing files where the allocated size
greatly differs from the total size:

$ ls -n --si -s sid-amd64.img
694M -rw-rw---- 1 1000 1000 27G Jan 27 17:48 sid-amd64.img

I am more used to seeing files where the two sizes are approximately
equal:

$ ls -n --si -s sid-autopkgtest-amd64.qcow2
950M -rw-r--r-- 1 1000 1000 950M Dec 30 17:55 sid-autopkgtest-amd64.qcow2

Not exactly equal, agreed:

$ ls -n -s sid-autopkgtest-amd64.qcow2
927604 -rw-r--r-- 1 1000 1000 949878784 Dec 30 17:55 sid-autopkgtest-amd64.qcow2
$ calc '927604*1024'
949866496

but very similar, indeed...

> Why would qcow images be less confusing?

Because it seems to me that their total and allocated sizes tend to be
more similar...

> Can you list some reasons why qcow2 should be preferred over
> raw images?

I was under the impression that the qcow2 format was the recommended
one for QEMU/KVM virtual machine images. At least, qemu-img(1)
describes it as "the most versatile format"...

Anyway, if you think that the raw format is a better choice for
autopkgtest and sbuild VM images, I take your word for it.

> Compression comes to mind but that is at the cost of doing that
> compression in the first place and it is not obvious to me whether cpu cycles
> or disk space are more scarce. For example, my platform is an ARM Cortex A53.
> To give you some idea how slow it is: it takes 30 seconds before a youtube
> video starts playing. At the same time, the system has a 2 TB hard disk. I much
> prefer a bigger disk image which needs less CPU when I want to create and use
> it. What is your use-case?

I just like to see smaller file sizes, whenever possible.
And I was speaking on the basis of recommendations read in QEMU
documentation, as I said.

But I understand that smaller files come at the cost of more
computation, if they are obtained through compression.
I agree with you that it is not obvious which aspect one should be
willing to sacrifice, in order to improve the other aspect.

>
> Thanks!

Thanks to you. :-)


--
http://www.inventati.org/frx/
There's not a second to spare! To the laboratory!
..................................................... Francesco Poli .
GnuPG key fpr == CA01 1147 9CD2 EFDF FB82 3925 3E1C 27E1 1F69 BFFE

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Francesco Poli (2024-01-29 00:07:08)

> What's *your* umask? Is it Debian default (022), by chance?

yes.

> This seems to work with Debian default umask (022):
>
> $ printf '%o\n' "$(( 0666 - 00022 ))"
> 644
>
> but fails whenever a umask includes an octal digit equal to 7, due to
> the carryover:
>
> $ printf '%o\n' "$(( 0666 - 00007 ))"
> 657
>
> Shouldn't this use the bitwise AND combined with the bitwise NOT?
>
> $ umask
> 0007
> $ printf '%o\n' "$(( 0666 & ~0$(umask) ))"
> 660
> $ umask 022
> $ umask
> 0022
> $ printf '%o\n' "$(( 0666 & ~0$(umask) ))"
> 644
>
> I would think that mmdebstrap-autopkgtest-build-qemu should run:
>
> chmod "$(printf %o "$(( 0666 & ~0$(umask) ))")" "$IMAGE"
>
> Does it make sense to you?

I now have this locally. Is this attribution (name/email) correct?

From 8410dc6636817c4e02cf9eba090a70051c494c48 Mon Sep 17 00:00:00 2001
From: Francesco Poli <inver...@paranoici.org>
Date: Mon, 29 Jan 2024 08:28:53 +0100
Subject: [PATCH] mmdebstrap-autopkgtest-build-qemu: fix octal mode computation

---
mmdebstrap-autopkgtest-build-qemu | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mmdebstrap-autopkgtest-build-qemu b/mmdebstrap-autopkgtest-build-qemu
index 5684cbe..1ed14db 100755
--- a/mmdebstrap-autopkgtest-build-qemu
+++ b/mmdebstrap-autopkgtest-build-qemu
@@ -357,7 +357,7 @@ echo "mmdebstrap $*"
mmdebstrap "$@" || die "mmdebstrap failed"

unshare -U -r --map-groups=auto chown 0:0 "$IMAGE"
-chmod "$(printf %o "$(( 0666 - 0$(umask) ))")" "$IMAGE"
+chmod "$(printf %o "$(( 0666 & ~0$(umask) ))")" "$IMAGE"

echo "root=LABEL=autopkgtestvm rw console=ttyS0" > "$WORKDIR/cmdline"

--
2.39.2

> > Can you list some reasons why qcow2 should be preferred over
> > raw images?
>
> I was under the impression that the qcow2 format was the recommended
> one for QEMU/KVM virtual machine images. At least, qemu-img(1)
> describes it as "the most versatile format"...
>
> Anyway, if you think that the raw format is a better choice for
> autopkgtest and sbuild VM images, I take your word for it.

I'm asking because I'm not the expert. :)

I asked in #debian-devel and f_g (Fabian Grünbichler) commented that the main
reasoning for choosing qcow over raw images is snapshot, link clone and backing
image support.

I don't think these features are useful for users of autopkgtest or sbuild with
these images. Or do you see a use for them in this area?

Thanks!

cheers, josch

[Johannes Schauer Marin Rodrigues]

Hi,

On 2024-01-29 00:07, Francesco Poli wrote:
> I was under the impression that the qcow2 format was the recommended
> one for QEMU/KVM virtual machine images. At least, qemu-img(1)
> describes it as "the most versatile format"...
>
> Anyway, if you think that the raw format is a better choice for
> autopkgtest and sbuild VM images, I take your word for it.

as I said in my other mail, I'm not the expert on the topic, so I asked
some experts. :)

In the other mail I already paraphrased what f_g said. I now also got
feedback from mjt (Michael Tokarev) our QEMU maintainer in Debian:

09:10 < mjt> for me, snapshots in qcow2 isn't of much use (I usually do
it the other way, by stacking another qcow2 on
top of the main image)
09:12 < jochensp> mjt: afair stacking only works if the base image is
already qcow2, or?
09:12 < mjt> speaking of qcow2 vs raw, both has their own good and bad
sides, mostly minor, and the preference is more
about taste than actual technical differences. People tend
to forget about sparseness of a raw image,
becoming surprizing the drivee usage increases dramatically
after a copy
09:13 < mjt> jochensp: no, stacking works on top of any image format
09:14 < mjt> ..on the other hand, trim support is more difficult on
qcow2
09:14 < mjt> myself, I choose raw for regular work and qcow2 when I have
to transfer the image somewhere
09:15 < mjt> it's quite easy to mount a qcow2 image on the host too, but
this needs extra layer (I usually use
qemu-nbd for this)
09:17 < mjt> for direct manipulation, when you have libext2fs and the
tools, raw is the only way to go

With that said, I think mmdebstrap-autopkgtest-build-qemu should keep
using raw images.

Thanks!

cheers, josch

[Francesco Poli]

On Mon, 29 Jan 2024 08:34:15 +0100 Johannes Schauer Marin Rodrigues wrote:

[...]

> I now have this locally. Is this attribution (name/email) correct?
>
> From 8410dc6636817c4e02cf9eba090a70051c494c48 Mon Sep 17 00:00:00 2001
> From: Francesco Poli <inver...@paranoici.org>
> Date: Mon, 29 Jan 2024 08:28:53 +0100
> Subject: [PATCH] mmdebstrap-autopkgtest-build-qemu: fix octal mode computation
>
> ---
> mmdebstrap-autopkgtest-build-qemu | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/mmdebstrap-autopkgtest-build-qemu b/mmdebstrap-autopkgtest-build-qemu
> index 5684cbe..1ed14db 100755
> --- a/mmdebstrap-autopkgtest-build-qemu
> +++ b/mmdebstrap-autopkgtest-build-qemu
> @@ -357,7 +357,7 @@ echo "mmdebstrap $*"
> mmdebstrap "$@" || die "mmdebstrap failed"
>
> unshare -U -r --map-groups=auto chown 0:0 "$IMAGE"
> -chmod "$(printf %o "$(( 0666 - 0$(umask) ))")" "$IMAGE"
> +chmod "$(printf %o "$(( 0666 & ~0$(umask) ))")" "$IMAGE"
>
> echo "root=LABEL=autopkgtestvm rw console=ttyS0" > "$WORKDIR/cmdline"
>
> --
> 2.39.2

Yes, the attribution looks correct.
Thanks for accepting the suggestion so promptly and for converting it
into an actual patch!

[Francesco Poli]

On Mon, 29 Jan 2024 10:21:29 +0100 Johannes Schauer Marin Rodrigues wrote:

[...]

> With that said, I think mmdebstrap-autopkgtest-build-qemu should keep
> using raw images.

OK, after reading these opinions, I can agree that it makes sense to go
on with raw images.
Thank you so much for taking the time to consult the experts!

Bye.

[Johannes Schauer Marin Rodrigues]

Quoting Francesco Poli (2024-01-29 20:54:25)

> On Mon, 29 Jan 2024 10:21:29 +0100 Johannes Schauer Marin Rodrigues wrote:
>
> [...]
> > With that said, I think mmdebstrap-autopkgtest-build-qemu should keep
> > using raw images.
>
> OK, after reading these opinions, I can agree that it makes sense to go
> on with raw images. Thank you so much for taking the time to consult the
> experts!

There is actually one good argument to go with qcow2: That is the default
format that autopkgtest-build-qemu uses. And since
mmdebstrap-autopkgtest-build-qemu tries to be like autopkgtest-build-qemu it
might make sense to choose the same output format...

But then I also thought: if the output format is raw, the user can always
easily run qemu-img afterwards if they want qcow2. The user can then even
decide for themselves if and what kind of compression or other features they
want. I think it is suboptimal that autopkgtest-build-qemu doesn not make this
optional and just enforces the step that converts to qcow2, eating up runtime
and cpu cycles even if the user doesn't require qcow2.

Since this question will come up again in the future, I added this list to the
code:

# The image is raw and not in qcow2 format because:
# - faster run-time as the "qemu-image convert" step is not needed
# - image can be used independent of qemu tooling
# - modifying the image just with "mount" instead of requiring qemu-nbd
# - sparse images make the file just as small as with qcow2
# - trim support is more difficult on qcow2
# - snapshots and overlays work just as well with raw images
# - users who prefer qcow2 get to choose to run it themselves with their own
# custom options like compression

Thanks!

cheers, josch