Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Francesco Poli (wintermute)]

Package: mmdebstrap
Version: 0.5.1-2
Severity: wishlist

Hello and thanks for developing/packaging this tool!

I wonder whether it can be used to create (without superuser privileges!)
a QEMU/KVM image.
I am especially interested in QEMU/KVM images suitable as autopkgtest
testbeds (autopkgtest-virt-qemu), but the feature could perhaps be
useful for building other minimal Debian base QEMU/KVM images as well...

As you most probably know, autopkgtest-build-qemu uses vmdb2 under the
hood, and vmdb2 [requires] to be run as root. I wonder whether mmdebstrap
can be used in stead of vmdb2, in order to lift the superuser privilege
requirement.

[requires]: <https://bugs.debian.org/944386>

Could this feature be implemented? It would really be awesome to have
a tool that allows a regular user to create a QEMU/KVM minimal Debian
image...

Please let me know your thoughts!
Thanks for your time and patience.
Bye.

[Johannes Schauer]

Hi Francesco,

Quoting Francesco Poli (wintermute) (2019-11-10 18:44:47)

> Hello and thanks for developing/packaging this tool!
>
> I wonder whether it can be used to create (without superuser privileges!)
> a QEMU/KVM image.
> I am especially interested in QEMU/KVM images suitable as autopkgtest
> testbeds (autopkgtest-virt-qemu), but the feature could perhaps be
> useful for building other minimal Debian base QEMU/KVM images as well...
>
> As you most probably know, autopkgtest-build-qemu uses vmdb2 under the
> hood, and vmdb2 [requires] to be run as root. I wonder whether mmdebstrap
> can be used in stead of vmdb2, in order to lift the superuser privilege
> requirement.
>
> [requires]: <https://bugs.debian.org/944386>
>
> Could this feature be implemented? It would really be awesome to have
> a tool that allows a regular user to create a QEMU/KVM minimal Debian
> image...

it does not need to be implemented because it is already possible.

It works through currently undocumented options that allow for hooks. Well,
actually the documentation already exists but is commented out, so you don't
see it in the man page that is generated from Perl POD. You can read the
documentation by reading the POD at the end of /usr/bin/mmdebstrap. For your
convenience I'll paste you the missing docs at the end of this mail. Part of
the docs is precisely what you were asking for: how to use mmdebstrap to
replace autopkgtest-build-qemu.

Thanks!

cheers, josch


--setup-hook=command
Execute arbitrary commands right after initial setup (directory creation,
configuration of apt and dpkg, ...) but before any packages are downloaded
or installed. At that point, the chroot directory does not contain any
executables and thus cannot be chroot-ed into. The option can be
specified multiple times and the commands are executed in the order in
which they are given on the command line. If command is an existing
executable file or if command does not contain any shell metacharacters,
then command is directly exec-ed with the path to the chroot directory
passed as the first argument. Otherwise, command is executed under sh and
the chroot directory can be accessed via $1. All environment variables
used by mmdebstrap (like "APT_CONFIG", "DEBIAN_FRONTEND", "LC_ALL" and
"PATH") are preserved.

Example: Setup merged-/usr via symlinks

--setup-hook='for d in bin sbin lib; do ln -s usr/$d "$1/$d"; mkdir -p "$1/usr/$d"; done'

Example: Setup chroot for installing a sub-essential busybox-based chroot
with --variant=custom
--include=dpkg,busybox,libc-bin,base-files,base-passwd,debianutils

--setup-hook='mkdir -p "$1/bin"'
--setup-hook='for p in awk cat chmod chown cp diff echo env grep less ln mkdir mount rm rmdir sed sh sleep sort touch uname; do ln -s busybox "$1/bin/$p"; done'
--setup-hook='echo root:x:0:0:root:/root:/bin/sh > "$1/etc/passwd"'
--setup-hook='printf "root:x:0:\nmail:x:8:\nutmp:x:43:\n" > "$1/etc/group"'

--essential-hook=command
Execute arbitrary commands after the Essential:yes packages have been
installed but before installing the remaining packages. The hook is not
executed for the extract and custom variants. The option can be specified
multiple times and the commands are executed in the order in which they
are given on the command line. If command is an existing executable file
or if command does not contain any shell metacharacters, then command is
directly exec-ed with the path to the chroot directory passed as the first
argument. Otherwise, command is executed under sh and the chroot directory
can be accessed via $1. All environment variables used by mmdebstrap (like
"APT_CONFIG", "DEBIAN_FRONTEND", "LC_ALL" and "PATH") are preserved.

Example: Enable unattended upgrades

--essential-hook='echo unattended-upgrades unattended-upgrades/enable_auto_updates boolean true | chroot "$1" debconf-set-selections'

Example: Select Europe/Berlin as the timezone

--essential-hook='echo tzdata tzdata/Areas select Europe | chroot "$1" debconf-set-selections'
--essential-hook='echo tzdata tzdata/Zones/Europe select Berlin | chroot "$1" debconf-set-selections'
--customize-hook=command
Execute arbitrary commands after the chroot is set up and all packages got
installed but before final cleanup actions are carried out. The option
can be specified multiple times and the commands are executed in the order
in which they are given on the command line. If command is an existing
executable file or if command does not contain any shell metacharacters,
then command is directly exec-ed with the path to the chroot directory
passed as the first argument. Otherwise, command is executed under sh and
the chroot directory can be accessed via $1. All environment variables
used by mmdebstrap (like "APT_CONFIG", "DEBIAN_FRONTEND", "LC_ALL" and
"PATH") are preserved.

Example: Preparing a chroot for use with autopkgtest

--customize-hook='chroot "$1" passwd --delete root'
--customize-hook='chroot "$1" useradd --home-dir /home/user --create-home user'
--customize-hook='chroot "$1" passwd --delete user'
--customize-hook='echo host > "$1/etc/hostname"'
--customize-hook='echo "127.0.0.1 localhost host" > "$1/etc/hosts"'
--customize-hook=/usr/share/autopkgtest/setup-commands/setup-testbed

Use as replacement for autopkgtest-build-qemu and vmdb2:

$ mmdebstrap --variant=important --include=linux-image-amd64 \
--customize-hook='chroot "$1" passwd --delete root' \
--customize-hook='chroot "$1" useradd --home-dir /home/user --create-home user' \
--customize-hook='chroot "$1" passwd --delete user' \
--customize-hook='echo host > "$1/etc/hostname"' \
--customize-hook='echo "127.0.0.1 localhost host" > "$1/etc/hosts"' \
--customize-hook=/usr/share/autopkgtest/setup-commands/setup-testbed \
unstable debian-unstable.tar
$ cat << END > extlinux.conf
> default linux
> timeout 0
>
> label linux
> kernel /vmlinuz
> append initrd=/initrd.img root=/dev/vda1 rw console=ttyS0
END
$ guestfish -N debian-unstable.img=disk:2G -- \
part-disk /dev/sda mbr : \
part-set-bootable /dev/sda 1 true : \
mkfs ext2 /dev/sda1 : mount /dev/sda1 / : \
tar-in debian-unstable.tar / : \
extlinux / : \
copy-in extlinux.conf /
$ qemu-img convert -O qcow2 debian-unstable.img debian-unstable.qcow2

[Francesco Poli]

On Sun, 10 Nov 2019 23:17:04 +0100 Johannes Schauer wrote:

[...]

> Quoting Francesco Poli (wintermute) (2019-11-10 18:44:47)

[...]

> > Could this feature be implemented? It would really be awesome to have
> > a tool that allows a regular user to create a QEMU/KVM minimal Debian
> > image...
>
> it does not need to be implemented because it is already possible.

[...]

Wow! Thanks for your super-prompt reply! :-)
And above all, thanks for implementing this already!

I think I will look into mmdebstrap in more depth and then get back to
you to let you know how it went...


--
http://www.inventati.org/frx/
There's not a second to spare! To the laboratory!
..................................................... Francesco Poli .
GnuPG key fpr == CA01 1147 9CD2 EFDF FB82 3925 3E1C 27E1 1F69 BFFE

[Francesco Poli]

On Mon, 11 Nov 2019 00:22:46 +0100 Francesco Poli wrote:

[...]

> I think I will look into mmdebstrap in more depth and then get back to
> you to let you know how it went...

I am giving it a try, but I am getting an error I am not quite sure to
understand:

$ mmdebstrap --variant=important --include=linux-image-amd64 \
--customize-hook='chroot "$1" passwd --delete root' \
--customize-hook='chroot "$1" useradd --home-dir /home/user --create-home user' \
--customize-hook='chroot "$1" passwd --delete user' \
--customize-hook='echo host > "$1/etc/hostname"' \
--customize-hook='echo "127.0.0.1 localhost host" > "$1/etc/hosts"' \
--customize-hook="/usr/share/autopkgtest/setup-commands/setup-testbed" \

sid debian-unstable.tar
I: automatically chosen mode: fakechroot
I: chroot architecture amd64 is equal to the host's architecture
[...]
Setting up linux-image-5.3.0-2-amd64 (5.3.9-2) ...
I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.3.0-2-amd64
I: /initrd.img.old is now a symlink to boot/initrd.img-5.3.0-2-amd64
I: /vmlinuz is now a symlink to boot/vmlinuz-5.3.0-2-amd64
I: /initrd.img is now a symlink to boot/initrd.img-5.3.0-2-amd64
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-5.3.0-2-amd64
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8107e-2.fw for module r8169
[...]
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8168d-1.fw for module r8169
E: /usr/share/initramfs-tools/hooks/udev failed with return 1.
update-initramfs: failed for /boot/initrd.img-5.3.0-2-amd64 with 1.
run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1
dpkg: error processing package linux-image-5.3.0-2-amd64 (--configure):
installed linux-image-5.3.0-2-amd64 package post-installation script subprocess returned error exit status 1
dpkg: dependency problems prevent configuration of linux-image-amd64:
linux-image-amd64 depends on linux-image-5.3.0-2-amd64 (= 5.3.9-2); however:
Package linux-image-5.3.0-2-amd64 is not configured yet.

dpkg: error processing package linux-image-amd64 (--configure):
dependency problems - leaving unconfigured
Setting up tasksel (3.56) ...
Setting up tasksel-data (3.56) ...
Processing triggers for libc-bin (2.29-3) ...
Processing triggers for systemd (243-7) ...
Processing triggers for initramfs-tools (0.135) ...
Errors were encountered while processing:
linux-image-5.3.0-2-amd64
linux-image-amd64
E: Sub-process /usr/bin/dpkg returned an error code (1)
E: run_chroot failed: E: /usr/sbin/chroot /home/frx/var/cache/autopkgtest_TEST1/mmdebstrap.DUEBqVAD7B env --unset=APT_CONFIG --unset=TMPDIR apt-get --yes install -oAPT::Status-Fd=<$fd> -oDpkg::Use-Pty=false udev procps iproute2 ifupdown apt-utils mount tasksel iputils-ping systemd-sysv tzdata linux-image-amd64 debconf-i18n vim-common gcc-9-base iptables libpam-runtime e2fsprogs libpam-modules whiptail cron passwd isc-dhcp-common libpam-modules-bin init vim-tiny cpio netbase apt tasksel-data readline-common systemd kmod dmidecode logrotate sensible-utils gpgv mawk debconf bsdmainutils libreadline8 isc-dhcp-client nano debian-archive-keyring fdisk adduser less rsyslog failed
I: removing tempdir /tmp/mmdebstrap.DUEBqVAD7B...


I noticed that my /tmp was almost full, when mmdebstrap failed.
Since I have /tmp mounted on a physical partition:

$ df --si /tmp/
Filesystem Size Used Avail Use% Mounted on
/dev/md3 869M 934k 806M 1% /tmp

I thought it could be the cause of the issue.
I tried to set:

$ TMPDIR='./'
$ export TMPDIR

This forced mmdebstrap to create its temporary directory inside the
current working directory (which is inside my home directory, where at
least 150 GB are available), but did not solve the issue.

Could you please help me understand
why /usr/share/initramfs-tools/hooks/udev failed with exit status == 1 ?


Thanks for your time and patience!

[Johannes Schauer]

Hi,

Quoting Francesco Poli (2019-11-17 16:06:05)

> I am giving it a try, but I am getting an error I am not quite sure to
> understand:
>

> [...]

>
> I: automatically chosen mode: fakechroot

there is your problem. If you are not executing mmdebstrap with superuser
privileges and if you you did not manually specify which --mode to use, then it
will look for some way to create the tarball that works on your system. For you
it picked fakechroot. That mode works by using an LD_PRELOAD library to fake
root permissions for all executed programs. Sometimes this fails badly and the
postinst script of update-initramfs is a known maintainer script that doesn't
work under fakechroot mode. I briefly looked into the problem a few days ago
but couldn't immediately see what was wrong. So this is not a bug in
mmdebstrap. It is maybe a bug in fakechroot or maybe something could be done
different by the postinst script of update-initramfs. I don't know whether
either maintainer considers what you see a bug. You can try and file one and
see if you get this fixed somehow.

Another way to solve the problem is by choosing a different mode. If you don't
mind using superuser privileges, you could run mmdebstrap as root.
Alternatively, you could also try --mode=unshare which uses linux user
namespaces to avoid using root privileges to build your tarball.

Thanks!

cheers, josch

[Francesco Poli]

On Sun, 17 Nov 2019 16:17:25 +0100 Johannes Schauer wrote:

> Hi,

Hi, thanks again for replying so promptly!

>
> Quoting Francesco Poli (2019-11-17 16:06:05)
> > I am giving it a try, but I am getting an error I am not quite sure to
> > understand:
> >
> > [...]
> >
> > I: automatically chosen mode: fakechroot
>
> there is your problem. If you are not executing mmdebstrap with superuser
> privileges and if you you did not manually specify which --mode to use, then it
> will look for some way to create the tarball that works on your system. For you
> it picked fakechroot.

[...]

Well, I was trying the command you have previously suggested...

I understand that automatic mode depends on which packages I have
installed on my system and on my sysctl configuration.
The fact is: I was running as a regular user.
I have:

$ /sbin/sysctl kernel.unprivileged_userns_clone
kernel.unprivileged_userns_clone = 0

And I have fakechroot on my system, just because mmdebstrap recommends
it:

$ aptitude why fakechroot
i mmdebstrap Recommends fakechroot

Hence, everything conspired to make mmdebstrap choose that mode!

Maybe the recipe you suggested should be improved to address this issue
with the selection of the mode...

> That mode works by using an LD_PRELOAD library to fake
> root permissions for all executed programs. Sometimes this fails badly and the
> postinst script of update-initramfs is a known maintainer script that doesn't
> work under fakechroot mode. I briefly looked into the problem a few days ago
> but couldn't immediately see what was wrong. So this is not a bug in
> mmdebstrap. It is maybe a bug in fakechroot or maybe something could be done
> different by the postinst script of update-initramfs. I don't know whether
> either maintainer considers what you see a bug. You can try and file one and
> see if you get this fixed somehow.

I can try and report bugs, although I don't fully understand what's
going on, and hence I will need your help to explain. Are you willing
to join this bug reporting effort?

Anyway, I am not sure I understand how *you* create an autopkgtest
QEMU/KVM testbed with mmdebstrap. I thought you were successfully using
the recipe you suggested.
Is it not the case? How do *you* do that?

>
> Another way to solve the problem is by choosing a different mode. If you don't
> mind using superuser privileges, you could run mmdebstrap as root.

I really mind! ;-)
Creating QEMU/KVM images without superuser privileges was the whole
point of giving mmdebstrap a try.
Otherwise I would just be using autopkgtest-build-qemu, as I said in the
original bug report...

> Alternatively, you could also try --mode=unshare which uses linux user
> namespaces to avoid using root privileges to build your tarball.

This seems to require setting "kernel.unprivileged_userns_clone=1" with
sysctl. But, after having a look at #898446, I am under the impression
that doing so would reduce the security of my system...

Any other possibility to create autopkgtest QEMU/KVM images without
root privileges?

[Johannes Schauer]

Hi,

Quoting Francesco Poli (2019-11-17 16:56:48)

> > Quoting Francesco Poli (2019-11-17 16:06:05)
> > > I am giving it a try, but I am getting an error I am not quite sure to
> > > understand:
> > >
> > > [...]
> > >
> > > I: automatically chosen mode: fakechroot
> > there is your problem. If you are not executing mmdebstrap with superuser
> > privileges and if you you did not manually specify which --mode to use,
> > then it will look for some way to create the tarball that works on your
> > system. For you it picked fakechroot.
> [...]
>
> Well, I was trying the command you have previously suggested...
>
> I understand that automatic mode depends on which packages I have
> installed on my system and on my sysctl configuration.

Yes, correct.

> The fact is: I was running as a regular user.
> I have:
>
> $ /sbin/sysctl kernel.unprivileged_userns_clone
> kernel.unprivileged_userns_clone = 0
>
> And I have fakechroot on my system, just because mmdebstrap recommends
> it:
>
> $ aptitude why fakechroot
> i mmdebstrap Recommends fakechroot
>
> Hence, everything conspired to make mmdebstrap choose that mode!

Also correct.

> Maybe the recipe you suggested should be improved to address this issue with
> the selection of the mode...

Before doing that I'll see if this could be fixed by changing something in
either fakechroot or in update-initramfs.

> > That mode works by using an LD_PRELOAD library to fake root permissions for
> > all executed programs. Sometimes this fails badly and the postinst script
> > of update-initramfs is a known maintainer script that doesn't work under
> > fakechroot mode. I briefly looked into the problem a few days ago but
> > couldn't immediately see what was wrong. So this is not a bug in
> > mmdebstrap. It is maybe a bug in fakechroot or maybe something could be
> > done different by the postinst script of update-initramfs. I don't know
> > whether either maintainer considers what you see a bug. You can try and
> > file one and see if you get this fixed somehow.
>
> I can try and report bugs, although I don't fully understand what's
> going on, and hence I will need your help to explain. Are you willing
> to join this bug reporting effort?

I'll take care of it. Before filing a bug I'll first see if I can maybe find
the root cause of it.

> Anyway, I am not sure I understand how *you* create an autopkgtest QEMU/KVM
> testbed with mmdebstrap. I thought you were successfully using the recipe you
> suggested. Is it not the case? How do *you* do that?

I see two possibilities:

1. back then when I wrote the docs a year ago, the bug in update-initramfs or
fakechroot might not've existed yet

2. maybe I had "kernel.unprivileged_userns_clone = 1" set and thus unshare
mode was selected during my testing

> > Another way to solve the problem is by choosing a different mode. If you don't
> > mind using superuser privileges, you could run mmdebstrap as root.
> I really mind! ;-)

Great! :D

> Creating QEMU/KVM images without superuser privileges was the whole
> point of giving mmdebstrap a try.
> Otherwise I would just be using autopkgtest-build-qemu, as I said in the
> original bug report...

You could also file a bug report against vmdb2 (which is used by
autopkgtest-build-qemu) and ask its maintainer whether they would consider
adding root-less operation to it. As shown by mmdebstrap it's possible to
create a qemu autopkgtest image without superuser privileges. Maybe other tools
can gain that functionality as well?

> > Alternatively, you could also try --mode=unshare which uses linux user
> > namespaces to avoid using root privileges to build your tarball.
>
> This seems to require setting "kernel.unprivileged_userns_clone=1" with
> sysctl. But, after having a look at #898446, I am under the impression
> that doing so would reduce the security of my system...

I cannot give you good advice about security.

But if all you are creating a Debian chroot and running only software from
Debian, then you are trusting that software already, no?

I mean, we are not talking about using user namespaces to contain a potentially
malicious proprietary application but about the same code which otherwise
already is running with superuser privileges on millions of systems.

> Any other possibility to create autopkgtest QEMU/KVM images without root
> privileges?

You could also try using --mode=proot.

Thanks!

cheers, josch

[Johannes Schauer]

Quoting Johannes Schauer (2019-11-17 18:20:17)

> > Maybe the recipe you suggested should be improved to address this issue
> > with the selection of the mode...
>
> Before doing that I'll see if this could be fixed by changing something in
> either fakechroot or in update-initramfs.

See #944929

[Francesco Poli]

Hello Johannes!

After seeing bug #944929 closed as fixed in both unstable and testing,
I thought I could retry with mmdebstrap.
But I still get an error:

$ cd Downloads

$ TMPDIR='./'
$ export TMPDIR

$ mmdebstrap --variant=important --include=linux-image-amd64 \
> --customize-hook='chroot "$1" passwd --delete root' \
> --customize-hook='chroot "$1" useradd --home-dir /home/user --create-home user' \
> --customize-hook='chroot "$1" passwd --delete user' \
> --customize-hook='echo host > "$1/etc/hostname"' \
> --customize-hook='echo "127.0.0.1 localhost host" > "$1/etc/hosts"' \
> --customize-hook="/usr/share/autopkgtest/setup-commands/setup-testbed" \
> "sid" debian-unstable.tar

I: automatically chosen mode: fakechroot

I: chroot architecture amd64 is equal to the host's architecture

I: using ${HOME}/Downloads/mmdebstrap.2GP5UNhTac as tempdir
I: running apt-get update...
done
I: downloading packages with apt...
done
I: extracting archives...
done
I: installing packages...
done
I: downloading apt...
done
I: installing apt...
done
I: installing remaining packages inside the chroot...
done
I: running --customize-hook in shell: sh -c 'chroot "$1" passwd --delete root' exec ${HOME}/Downloads/mmdebstrap.2GP5UNhTac
passwd: password expiry information changed.
I: running --customize-hook in shell: sh -c 'chroot "$1" useradd --home-dir /home/user --create-home user' exec ${HOME}/Downloads/mmdebstrap.2GP5UNhTac
I: running --customize-hook in shell: sh -c 'chroot "$1" passwd --delete user' exec ${HOME}/Downloads/mmdebstrap.2GP5UNhTac
passwd: password expiry information changed.
I: running --customize-hook in shell: sh -c 'echo host > "$1/etc/hostname"' exec ${HOME}/Downloads/mmdebstrap.2GP5UNhTac
I: running --customize-hook in shell: sh -c 'echo "127.0.0.1 localhost host" > "$1/etc/hosts"' exec ${HOME}/Downloads/mmdebstrap.2GP5UNhTac
I: running --customize-hook directly: /usr/share/autopkgtest/setup-commands/setup-testbed ${HOME}/Downloads/mmdebstrap.2GP5UNhTac
/usr/share/autopkgtest/setup-commands/setup-testbed: Attempting to set up Debian/Ubuntu apt sources automatically
/usr/share/autopkgtest/setup-commands/setup-testbed: Distribution assumed to resemble Debian
update-initramfs: Generating /boot/initrd.img-5.4.0-3-amd64
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8125a-3.fw for module r8169

[...]
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8168d-1.fw for module r8169

cp: failed to access './/mkinitramfs_SqDsE7/lib/systemd/network/': No such file or directory

E: /usr/share/initramfs-tools/hooks/udev failed with return 1.

update-initramfs: failed for /boot/initrd.img-5.4.0-3-amd64 with 1.
E: run_chroot failed: E: command failed: /usr/share/autopkgtest/setup-commands/setup-testbed
I: main() received signal PIPE: waiting for setup...
W: listening on child socket failed: E: received eof on socket

I: removing tempdir ${HOME}/Downloads/mmdebstrap.2GP5UNhTac...
$ ls -s
total 0
0 debian-unstable.tar

As you can see, I still obtain an empty file.

Did I misunderstand the situation?
I thought it could work correctly, now...

[Johannes Schauer]

Hi,

Quoting Francesco Poli (2020-01-27 23:24:22)

> I: running --customize-hook directly: /usr/share/autopkgtest/setup-commands/setup-testbed ${HOME}/Downloads/mmdebstrap.2GP5UNhTac
> /usr/share/autopkgtest/setup-commands/setup-testbed: Attempting to set up Debian/Ubuntu apt sources automatically
> /usr/share/autopkgtest/setup-commands/setup-testbed: Distribution assumed to resemble Debian
> update-initramfs: Generating /boot/initrd.img-5.4.0-3-amd64
> W: Possible missing firmware /lib/firmware/rtl_nic/rtl8125a-3.fw for module r8169
> [...]
> W: Possible missing firmware /lib/firmware/rtl_nic/rtl8168d-1.fw for module r8169
> cp: failed to access './/mkinitramfs_SqDsE7/lib/systemd/network/': No such file or directory
> E: /usr/share/initramfs-tools/hooks/udev failed with return 1.
> update-initramfs: failed for /boot/initrd.img-5.4.0-3-amd64 with 1.
> E: run_chroot failed: E: command failed: /usr/share/autopkgtest/setup-commands/setup-testbed
> I: main() received signal PIPE: waiting for setup...
> W: listening on child socket failed: E: received eof on socket
>
> I: removing tempdir ${HOME}/Downloads/mmdebstrap.2GP5UNhTac...
> $ ls -s
> total 0
> 0 debian-unstable.tar
>
> As you can see, I still obtain an empty file.
>
> Did I misunderstand the situation?
> I thought it could work correctly, now...

this looks as if the error comes from
/usr/share/autopkgtest/setup-commands/setup-testbed. To figure out what goes
wrong, maybe try running setup-testbed with sh -x like so:

--customize-hook='sh -x /usr/share/autopkgtest/setup-commands/setup-testbed "$1"'

Your command works fine on my system, so this must be something specific to
your setup.

Thanks!

cheers, josch

[Francesco Poli]

On Mon, 27 Jan 2020 23:44:55 +0100 Johannes Schauer wrote:

[...]

> this looks as if the error comes from
> /usr/share/autopkgtest/setup-commands/setup-testbed. To figure out what goes
> wrong, maybe try running setup-testbed with sh -x like so:
>
> --customize-hook='sh -x /usr/share/autopkgtest/setup-commands/setup-testbed "$1"'
>
> Your command works fine on my system, so this must be something specific to
> your setup.
>
> Thanks!

Thanks to you for your super-prompt reply! :-)

$ cd Downloads/

$ TMPDIR='./'
$ export TMPDIR
$ mmdebstrap --variant=important --include=linux-image-amd64 \
> --customize-hook='chroot "$1" passwd --delete root' \
> --customize-hook='chroot "$1" useradd --home-dir /home/user --create-home user' \
> --customize-hook='chroot "$1" passwd --delete user' \
> --customize-hook='echo host > "$1/etc/hostname"' \
> --customize-hook='echo "127.0.0.1 localhost host" > "$1/etc/hosts"' \

> --customize-hook='sh -x /usr/share/autopkgtest/setup-commands/setup-testbed "$1"' \

> "sid" debian-unstable.tar
I: automatically chosen mode: fakechroot

[...]
I: running --customize-hook in shell: sh -c 'sh -x /usr/share/autopkgtest/setup-commands/setup-testbed "$1"' exec ${HOME}/Downloads/mmdebstrap.pxni4vAjou
+ set -eu
+ umask 0022
+ export DEBIAN_FRONTEND=noninteractive
+ [ ${HOME}/Downloads/mmdebstrap.pxni4vAjou = --help ]
+ root=${HOME}/Downloads/mmdebstrap.pxni4vAjou
+ [ ${HOME}/Downloads/mmdebstrap.pxni4vAjou != / ]
+ cat
+ chmod 755 ${HOME}/Downloads/mmdebstrap.pxni4vAjou/etc/init.d/autopkgtest
+ chroot ${HOME}/Downloads/mmdebstrap.pxni4vAjou update-rc.d autopkgtest defaults
+ [ -d ${HOME}/Downloads/mmdebstrap.pxni4vAjou/etc/systemd/system ]
+ cat
+ mkdir -p ${HOME}/Downloads/mmdebstrap.pxni4vAjou/etc/systemd/system/multi-user.target.wants
+ ln -sf ../autopkgtest.service ${HOME}/Downloads/mmdebstrap.pxni4vAjou/etc/systemd/system/multi-user.target.wants/autopkgtest.service
+ [ -e ${HOME}/Downloads/mmdebstrap.pxni4vAjou/etc/init/tty2.conf -a ! -e ${HOME}/Downloads/mmdebstrap.pxni4vAjou/etc/init/ttyS0.conf ]
+ chroot ${HOME}/Downloads/mmdebstrap.pxni4vAjou dpkg --print-architecture
+ ARCH=amd64
+ [ ! -e ${HOME}/Downloads/mmdebstrap.pxni4vAjou/etc/default/grub.d/90-autopkgtest.cfg ]
+ chroot ${HOME}/Downloads/mmdebstrap.pxni4vAjou which update-grub
+ [ -e ${HOME}/Downloads/mmdebstrap.pxni4vAjou/etc/os-release ]
+ . ${HOME}/Downloads/mmdebstrap.pxni4vAjou/etc/os-release
+ PRETTY_NAME=Debian GNU/Linux bullseye/sid
+ NAME=Debian GNU/Linux
+ ID=debian
+ HOME_URL=https://www.debian.org/
+ SUPPORT_URL=https://www.debian.org/support
+ BUG_REPORT_URL=https://bugs.debian.org/
+ echo debian
+ DISTRO_ID=debian
+ [ -z ]
+ awk /^deb .*debian/ { sub(/\[.*\]/, "", $0); print $2; exit } ${HOME}/Downloads/mmdebstrap.pxni4vAjou/etc/apt/sources.list
+ MIRROR=http://deb.debian.org/debian
+ [ -z ]
+ awk /^deb .*debian/ { sub(/\[.*\]/, "", $0); print $3; exit } ${HOME}/Downloads/mmdebstrap.pxni4vAjou/etc/apt/sources.list
+ RELEASE=sid
+ [ -n ]
+ [ -n ]
+ [ -n ]
+ echo /usr/share/autopkgtest/setup-commands/setup-testbed: Attempting to set up Debian/Ubuntu apt sources automatically

/usr/share/autopkgtest/setup-commands/setup-testbed: Attempting to set up Debian/Ubuntu apt sources automatically

+ [ -z sid ]
+ [ -z http://deb.debian.org/debian ]
+ [ http://deb.debian.org/debian != http://deb.debian.org/debian ]
+ echo /usr/share/autopkgtest/setup-commands/setup-testbed: Distribution assumed to resemble Debian

/usr/share/autopkgtest/setup-commands/setup-testbed: Distribution assumed to resemble Debian

+ cat
+ [ -e ${HOME}/Downloads/mmdebstrap.pxni4vAjou/etc/cloud/cloud.cfg ]
+ [ -z ]
+ ls ${HOME}/Downloads/mmdebstrap.pxni4vAjou/etc/systemd/network/*.network
+ ls ${HOME}/Downloads/mmdebstrap.pxni4vAjou/etc/netplan/*.yaml
+ grep -q source.*interfaces.d ${HOME}/Downloads/mmdebstrap.pxni4vAjou/etc/network/interfaces
+ IFACE=
+ [ ${HOME}/Downloads/mmdebstrap.pxni4vAjou = / ]
+ IFACE=eth0
+ [ -e ${HOME}/Downloads/mmdebstrap.pxni4vAjou/etc/udev/rules.d/80-net-setup-link.rules ]
+ ln -s /dev/null ${HOME}/Downloads/mmdebstrap.pxni4vAjou/etc/udev/rules.d/80-net-setup-link.rules
+ chroot ${HOME}/Downloads/mmdebstrap.pxni4vAjou update-initramfs -u

update-initramfs: Generating /boot/initrd.img-5.4.0-3-amd64
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8125a-3.fw for module r8169
[...]
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8168d-1.fw for module r8169

cp: failed to access './/mkinitramfs_v8SvPb/lib/systemd/network/': No such file or directory

E: /usr/share/initramfs-tools/hooks/udev failed with return 1.
update-initramfs: failed for /boot/initrd.img-5.4.0-3-amd64 with 1.

E: run_chroot failed: E: command failed: sh -x /usr/share/autopkgtest/setup-commands/setup-testbed "$1"

I: main() received signal PIPE: waiting for setup...
W: listening on child socket failed: E: received eof on socket

I: removing tempdir ${HOME}/Downloads/mmdebstrap.pxni4vAjou...


As you can see, a "cp" fails to access file './/mkinitramfs_v8SvPb/lib/systemd/network/',
and /usr/share/initramfs-tools/hooks/udev fails with exit status 1.
But, honestly, I cannot understand why.

The lines that seem to fail are /usr/share/initramfs-tools/hooks/udev:
25 and the following ones... But why?

[Johannes Schauer]

Hi,

Quoting Francesco Poli (2020-01-28 23:41:38)

> + chroot ${HOME}/Downloads/mmdebstrap.pxni4vAjou update-initramfs -u
> update-initramfs: Generating /boot/initrd.img-5.4.0-3-amd64
> W: Possible missing firmware /lib/firmware/rtl_nic/rtl8125a-3.fw for module r8169
> [...]
> W: Possible missing firmware /lib/firmware/rtl_nic/rtl8168d-1.fw for module r8169
> cp: failed to access './/mkinitramfs_v8SvPb/lib/systemd/network/': No such file or directory
> E: /usr/share/initramfs-tools/hooks/udev failed with return 1.
> update-initramfs: failed for /boot/initrd.img-5.4.0-3-amd64 with 1.
> E: run_chroot failed: E: command failed: sh -x /usr/share/autopkgtest/setup-commands/setup-testbed "$1"
> I: main() received signal PIPE: waiting for setup...
> W: listening on child socket failed: E: received eof on socket
>
> I: removing tempdir ${HOME}/Downloads/mmdebstrap.pxni4vAjou...
>
>
> As you can see, a "cp" fails to access file './/mkinitramfs_v8SvPb/lib/systemd/network/',
> and /usr/share/initramfs-tools/hooks/udev fails with exit status 1.
> But, honestly, I cannot understand why.
>
> The lines that seem to fail are /usr/share/initramfs-tools/hooks/udev:
> 25 and the following ones... But why?

yes, that's where it seems to fail. To further investigate you can try the
following:

Copy /usr/share/autopkgtest/setup-commands/setup-testbed to a location that you
control so that you can edit it. Remove all the parts that are not necessary to
reproduce the problem. I suspect just keeping the line 'chroot "$root"
update-initramfs -u' is not sufficient and something else has to happen before
so that you can see the error? If that line alone is enough, then you should
also be able to reproduce the problem by running mmdebstrap with:

--customize-hook='chroot "$1" update-initramfs -u'

Once you are that far, you either already have found the problem or the next
thing you can try is to debug /usr/share/initramfs-tools/hooks/udev. To do
that, just insert a "set -x" somewhere at the top. For example by adding the
following before you call the failing hook:

--customize-hook='awk "NR==2{print 'set -x'}1" "$1/usr/share/initramfs-tools/hooks/udev"'

I would love to help more but I already tried out the failing command on three
different systems and I'm unable to reproduce it.

Thanks!

cheers, josch

[Francesco Poli]

On Wed, 29 Jan 2020 00:02:51 +0100 Johannes Schauer wrote:

[...]

> Quoting Francesco Poli (2020-01-28 23:41:38)

[...]

> > The lines that seem to fail are /usr/share/initramfs-tools/hooks/udev:
> > 25 and the following ones... But why?
>
> yes, that's where it seems to fail. To further investigate you can try the
> following:
>
> Copy /usr/share/autopkgtest/setup-commands/setup-testbed to a location that you
> control so that you can edit it. Remove all the parts that are not necessary to
> reproduce the problem. I suspect just keeping the line 'chroot "$root"
> update-initramfs -u' is not sufficient and something else has to happen before
> so that you can see the error? If that line alone is enough, then you should
> also be able to reproduce the problem by running mmdebstrap with:
>
> --customize-hook='chroot "$1" update-initramfs -u'

I can get the error with just:

$ cd Downloads/
$ TMPDIR='./'
$ export TMPDIR
$ mmdebstrap --variant=important --include=linux-image-amd64 \
> --customize-hook='chroot "$1" passwd --delete root' \
> --customize-hook='chroot "$1" useradd --home-dir /home/user --create-home user' \
> --customize-hook='chroot "$1" passwd --delete user' \
> --customize-hook='echo host > "$1/etc/hostname"' \
> --customize-hook='echo "127.0.0.1 localhost host" > "$1/etc/hosts"' \

> --customize-hook='chroot "$1" update-initramfs -u' \
> "sid" debian-unstable.tar

>
> Once you are that far, you either already have found the problem or the next
> thing you can try is to debug /usr/share/initramfs-tools/hooks/udev. To do
> that, just insert a "set -x" somewhere at the top. For example by adding the
> following before you call the failing hook:
>
> --customize-hook='awk "NR==2{print 'set -x'}1" "$1/usr/share/initramfs-tools/hooks/udev"'

I think I am getting nearer to understand the issue.


$ cd Downloads/
$ TMPDIR='.'

$ export TMPDIR
$ mmdebstrap --variant=important --include=linux-image-amd64 \
> --customize-hook='chroot "$1" passwd --delete root' \
> --customize-hook='chroot "$1" useradd --home-dir /home/user --create-home user' \
> --customize-hook='chroot "$1" passwd --delete user' \
> --customize-hook='echo host > "$1/etc/hostname"' \
> --customize-hook='echo "127.0.0.1 localhost host" > "$1/etc/hosts"' \

> --customize-hook='sed -i "1a set -x" "$1/usr/share/initramfs-tools/hooks/udev"' \
> --customize-hook='chroot "$1" update-initramfs -u' \
> "sid" debian-unstable.tar
[...]
+ PREREQS=
+ prereqs
+ echo
+ exit 0
+ PREREQS=
+ . /usr/share/initramfs-tools/hook-functions
+ mkdir -p ./mkinitramfs_1itDzt/lib/systemd
[...]
+ mkdir -p ./mkinitramfs_1itDzt/etc/udev
+ cp -p /etc/udev/udev.conf ./mkinitramfs_1itDzt/etc/udev/
+ mkdir -p ./mkinitramfs_1itDzt/lib/systemd/network/
+ find /lib/systemd/network -name *.link -execdir cp -pt ./mkinitramfs_1itDzt/lib/systemd/network/ {} +
cp: failed to access './mkinitramfs_1itDzt/lib/systemd/network/': No such file or directory

E: /usr/share/initramfs-tools/hooks/udev failed with return 1.
update-initramfs: failed for /boot/initrd.img-5.4.0-3-amd64 with 1.

E: run_chroot failed: E: command failed: chroot "$1" update-initramfs -u

I: main() received signal PIPE: waiting for setup...
W: listening on child socket failed: E: received eof on socket

I: removing tempdir ${HOME}/Downloads/mmdebstrap._p8KPoNDcS...

The issue seems to be that the -execdir option passed to find causes
the command "cp" to be run from the subdirectory containing the matched
file "/lib/systemd/network", but the destination directory is a
relative path "./mkinitramfs_1itDzt/lib/systemd/network/", which has
been previously created, but not under "/lib/systemd/network" !

If I set TMPDIR=$(pwd) in stead of TMPDIR='.' or TMPDIR='./'
I get to the end of the process without major errors.
Except for some warnings such as:

W: Unable to read ${HOME}/Downloads/48yluzcM8D - RealFileExists (2: No such file or directory)

Should I worry about them?!?

At least I obtain a non-empty debian-unstable.tar and I manage to
create debian-unstable.img with guestfish and then convert it to
debian-unstable.qcow2 with qemu-img...

Now I am running out of time, but I hope I will soon get to check the
resulting .qcow2 file (I have to test whether it can successfully be
used as an autopkgtest testbed).
I'll let you know, thanks for all you invaluable help so far! :-)

>
> I would love to help more but I already tried out the failing command on three
> different systems and I'm unable to reproduce it.

Which TMPDIR were you using?

[Johannes Schauer]

Hi,

Quoting Francesco Poli (2020-01-29 23:34:21)

> > I would love to help more but I already tried out the failing command on
> > three different systems and I'm unable to reproduce it.
> Which TMPDIR were you using?

apologies, I missed that you set TMPDIR. When I set it, then I can reproduce
your problem in an even more minimal fashion:

TMPDIR='./' mmdebstrap --include=linux-image-amd64 --customize-hook='chroot "$1" update-initramfs -u' sid debian-unstable.tar

I'll come back to you once I have fixed the issue.

Thanks!

cheers, josch

[Johannes Schauer]

Quoting Johannes Schauer (2020-01-30 00:19:44)

the problem is fixed when TMPDIR gets unset for hook. I'm now unsure how to
best fix the problem. I could generally unset TMPDIR for hooks or just update
the instructions like this:

$ mmdebstrap --variant=important --include=linux-image-amd64 \
--customize-hook='chroot "$1" passwd --delete root' \
--customize-hook='chroot "$1" useradd --home-dir /home/user --create-home user' \
--customize-hook='chroot "$1" passwd --delete user' \
--customize-hook='echo host > "$1/etc/hostname"' \
--customize-hook='echo "127.0.0.1 localhost host" > "$1/etc/hosts"' \

--customize-hook='env --unset=TMPDIR /usr/share/autopkgtest/setup-commands/setup-testbed "$1"' \
unstable debian-unstable.tar

hmm....

[Francesco Poli]

On Thu, 30 Jan 2020 01:26:09 +0100 Johannes Schauer wrote:

[...]

> the problem is fixed when TMPDIR gets unset for hook.

Wait, does this change the result?

I mean: some commands executed (directly or indirectly) by
setup-testbed perform operations on the directory which mmdebstrap set
up as TMPDIR.
If this is the case (as I think it is, at least for the udev hook
called by update-initramfs), I wonder whether unsetting TMPDIR causes
setup-testbed to perform some of its operations on the wrong directory...

Am I completely off-track?

[Johannes Schauer]

Hi,

Quoting Francesco Poli (2020-01-30 23:41:11)

> On Thu, 30 Jan 2020 01:26:09 +0100 Johannes Schauer wrote:
> > the problem is fixed when TMPDIR gets unset for hook.
> Wait, does this change the result?
>
> I mean: some commands executed (directly or indirectly) by
> setup-testbed perform operations on the directory which mmdebstrap set
> up as TMPDIR.
> If this is the case (as I think it is, at least for the udev hook
> called by update-initramfs), I wonder whether unsetting TMPDIR causes
> setup-testbed to perform some of its operations on the wrong directory...
>
> Am I completely off-track?

in your case, TMPDIR was a relative path. It doesn't even matter whether the
script reading TMPDIR runs inside or outside the chroot. That script might
change its working directory to anywhere during its operation. If it then tries
to access a file that expects to be in $TMPDIR it will obviously not find it
because it changed working directory.

Another scenario: imagine you set TMPDIR to an absolute path outside the
chroot. Naturally, that path is not available inside the chroot and programs
inside the chroot might fail when they try to store data into the non-existing
TMPDIR.

Your expectation was, that when you set TMPDIR, the only one who would consume
that environment variable would be mmdebstrap itself. I think that thought is
not unreasonable. That's why I think it's important that mmdebstrap unsets that
variable before it executes processes inside the chroot.

Thanks!

cheers, josch

[Francesco Poli]

On Fri, 31 Jan 2020 06:28:07 +0100 Johannes Schauer wrote:

[...]

> Quoting Francesco Poli (2020-01-30 23:41:11)

[...]

> > Wait, does this change the result?

[...]

> in your case, TMPDIR was a relative path.

[...]

> Your expectation was, that when you set TMPDIR, the only one who would consume
> that environment variable would be mmdebstrap itself. I think that thought is
> not unreasonable. That's why I think it's important that mmdebstrap unsets that
> variable before it executes processes inside the chroot.

All your reasoning makes sense to me.
I agree that TMPDIR should be unset by mmdebstrap before running hooks.

I tried to check that the resulting .tar file is not affected by this
TMPDIR unsetting.
I did the following:

$ cat ABSTMPDIR/test.sh
#!/bin/sh

SETUP_TESTBED="/usr/share/autopkgtest/setup-commands/setup-testbed"

LIMAVAR="amd64"
SUITE="sid"
SIZE="2GiB"

TMPDIR=$(pwd)
export TMPDIR
mmdebstrap --variant=important --include=linux-image-"$LIMAVAR" \

--customize-hook='chroot "$1" passwd --delete root' \
--customize-hook='chroot "$1" useradd --home-dir /home/user --create-home user' \
--customize-hook='chroot "$1" passwd --delete user' \
--customize-hook='echo host > "$1/etc/hostname"' \
--customize-hook='echo "127.0.0.1 localhost host" > "$1/etc/hosts"' \

--customize-hook="$SETUP_TESTBED" \
"$SUITE" debian-unstable.tar
$ cat NOTMPDIR/test.sh
#!/bin/sh

SETUP_TESTBED="/usr/share/autopkgtest/setup-commands/setup-testbed"

LIMAVAR="amd64"
SUITE="sid"
SIZE="2GiB"

TMPDIR='.'
export TMPDIR
mmdebstrap --variant=important --include=linux-image-"$LIMAVAR" \

--customize-hook='chroot "$1" passwd --delete root' \
--customize-hook='chroot "$1" useradd --home-dir /home/user --create-home user' \
--customize-hook='chroot "$1" passwd --delete user' \
--customize-hook='echo host > "$1/etc/hostname"' \
--customize-hook='echo "127.0.0.1 localhost host" > "$1/etc/hosts"' \
--customize-hook='env --unset=TMPDIR /usr/share/autopkgtest/setup-commands/setup-testbed "$1"' \

"$SUITE" debian-unstable.tar

After running the two test.sh scripts within their respective
directories, I used diffoscope to highlight the differences between the
two resulting .tar files:

$ TMPDIR=/run/shm diffoscope \
--max-diff-block-lines 15000 \
--max-page-diff-block-lines 1500 \
--html report_diffoscope.html \
ABSTMPDIR/debian-unstable.tar \
NOTMPDIR/debian-unstable.tar

The only differences shown in the resulting report_diffoscope.html file
seem to be:

• the generated files in the content
of ./boot/initrd.img-5.4.0-3-amd64 have differing creation
timestamps (but this is obvious, since the two initrd were not
created exactly at the same time!)

• ./var/lib/initramfs-tools/5.4.0-3-amd64 files differ (but they seem
to include some checksum of the initrd, hence the difference should
be consequence of the first point)

• ./etc/machine-id and ./var/lib/dbus/machine-id files differ (but I
think this should not be surprising...)

• ./usr/lib/python3.7/__pycache__/hashlib.cpython-37.pyc files have
some different hex values (I am not sure why, but it's compiled
Python code, maybe it includes a compilation timestamp or
something?!?)

I am under the impression that the two .tar files are to be considered
equivalent.
Do you agree?



P.S.: I still have to find the time to check the .qcow2 file I obtained
on last Wednesday... :-(

[Johannes Schauer]

Quoting Francesco Poli (2020-02-01 16:52:47)

> The only differences shown in the resulting report_diffoscope.html file seem
> to be:
>
> • the generated files in the content
> of ./boot/initrd.img-5.4.0-3-amd64 have differing creation
> timestamps (but this is obvious, since the two initrd were not
> created exactly at the same time!)
>
> • ./var/lib/initramfs-tools/5.4.0-3-amd64 files differ (but they seem
> to include some checksum of the initrd, hence the difference should
> be consequence of the first point)

this should go away when you set SOURCE_DATE_EPOCH to something like $(date
+%s) -- why didn't you set it in your tests?

> • ./etc/machine-id and ./var/lib/dbus/machine-id files differ (but I
> think this should not be surprising...)

In the next mmdebstrap release /etc/machine-id will be set to an empty file.

> • ./usr/lib/python3.7/__pycache__/hashlib.cpython-37.pyc files have
> some different hex values (I am not sure why, but it's compiled
> Python code, maybe it includes a compilation timestamp or
> something?!?)

This is a known bug that I have yet to report to the Python maintainers.

> I am under the impression that the two .tar files are to be considered
> equivalent.
> Do you agree?

Yes. :)

[Francesco Poli]

On Sun, 02 Feb 2020 00:23:59 +0100 Johannes Schauer wrote:

> Quoting Francesco Poli (2020-02-01 16:52:47)
> > The only differences shown in the resulting report_diffoscope.html file seem
> > to be:
> >
> > • the generated files in the content
> > of ./boot/initrd.img-5.4.0-3-amd64 have differing creation
> > timestamps (but this is obvious, since the two initrd were not
> > created exactly at the same time!)
> >
> > • ./var/lib/initramfs-tools/5.4.0-3-amd64 files differ (but they seem
> > to include some checksum of the initrd, hence the difference should
> > be consequence of the first point)
>
> this should go away when you set SOURCE_DATE_EPOCH to something like $(date
> +%s) -- why didn't you set it in your tests?

It's plain simple: because I am an idiot! ;-)
I just failed to think about it...

After setting the same SOURCE_DATE_EPOCH, the only remaining
differences were the machine-id and the hashlib.cpython-37.pyc file!

>
> > • ./etc/machine-id and ./var/lib/dbus/machine-id files differ (but I
> > think this should not be surprising...)
>
> In the next mmdebstrap release /etc/machine-id will be set to an empty file.

OK.

>
> > • ./usr/lib/python3.7/__pycache__/hashlib.cpython-37.pyc files have
> > some different hex values (I am not sure why, but it's compiled
> > Python code, maybe it includes a compilation timestamp or
> > something?!?)
>
> This is a known bug that I have yet to report to the Python maintainers.

Ah, interesting.
I encourage you to report this bug, as it might help the Reproducible
Builds effort...

>
> > I am under the impression that the two .tar files are to be considered
> > equivalent.
> > Do you agree?
>
> Yes. :)

OK, in the meanwhile I got around to check whether the .qcow2 image is
actually working as autopkgtest testbed.

Unfortunately, no, it's not working! :-(


$ autopkgtest --output-dir ./${PKG}_${VERS}_autopkgtest.out \
--summary ./${PKG}_${VERS}_autopkgtest.summary \
-B ./${PKG}_${VERS}_amd64.changes \
-- qemu ~/Downloads/TEST/debian-unstable.qcow2
autopkgtest [16:23:45]: version 5.11
autopkgtest [16:23:45]: host ${HOST}; command line: ${CMDLINE}
qemu-system-x86_64: terminating on signal 15 from pid 8488 (/usr/bin/python3)
<VirtSubproc>: failure: timed out waiting for "login prompt on ttyS0"
autopkgtest [16:24:45]: ERROR: testbed failure: cannot send to testbed: [Errno 32] Broken pipe

Could you help me to investigate the issue?
Is the command line correct?
Where did I go wrong?

[Johannes Schauer]

Hi,

Quoting Francesco Poli (2020-02-02 16:58:05)

> > > • ./usr/lib/python3.7/__pycache__/hashlib.cpython-37.pyc files have
> > > some different hex values (I am not sure why, but it's compiled Python
> > > code, maybe it includes a compilation timestamp or something?!?)
> > This is a known bug that I have yet to report to the Python maintainers.
> Ah, interesting. I encourage you to report this bug, as it might help the
> Reproducible Builds effort...

I reported a very similar bug over a year ago:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=917407

The bug then magically fixed itself when python 3.7.3 upgraded to 3.7.4 but
nobody really ever investigated what happened.

> > > I am under the impression that the two .tar files are to be considered
> > > equivalent. Do you agree?
> > Yes. :)
> OK, in the meanwhile I got around to check whether the .qcow2 image is
> actually working as autopkgtest testbed.
>
> Unfortunately, no, it's not working! :-(
>
>
> $ autopkgtest --output-dir ./${PKG}_${VERS}_autopkgtest.out \
> --summary ./${PKG}_${VERS}_autopkgtest.summary \
> -B ./${PKG}_${VERS}_amd64.changes \
> -- qemu ~/Downloads/TEST/debian-unstable.qcow2
> autopkgtest [16:23:45]: version 5.11
> autopkgtest [16:23:45]: host ${HOST}; command line: ${CMDLINE}
> qemu-system-x86_64: terminating on signal 15 from pid 8488 (/usr/bin/python3)
> <VirtSubproc>: failure: timed out waiting for "login prompt on ttyS0"
> autopkgtest [16:24:45]: ERROR: testbed failure: cannot send to testbed: [Errno 32] Broken pipe
>
> Could you help me to investigate the issue?
> Is the command line correct? Where did I go wrong?

I can imagine two ways forward.

1. you can try the qcow2 image you built without autopkgtest by just running it
inside qemu like so:

$ qemu-system-x86_64 -enable-kvm -m 512 -serial unix:/tmp/ttyS0,server,nowait -drive "file=$HOME/Downloads/TEST/debian-unstable.qcow2,cache=unsafe,if=virtio,index=0"

2. you could upload your qcow2 image somewhere together with the right
SOURCE_DATE_EPOCH value that you used. Then I could try to reproduce that
image myself and diff it against what I have because what I have here is
working just fine. :)

Thanks!

cheers, josch

[Johannes Schauer]

Quoting Johannes Schauer (2020-02-02 20:22:49)

One non-obvious thing is missing here. After running the above command you have
a new unix socket under /tmp/ttyS0 that you should be able to connect to using
a serial terminal emulator. This is how autopkgtest connects to qemu. So after
running above command, check that everything works by running something like:

$ minicom -D 'unix#/tmp/ttyS0'

[Johannes Schauer]

Hi,

Quoting Francesco Poli (2020-02-17 23:10:57)

> On Sun, 02 Feb 2020 20:29:04 +0100 Johannes Schauer wrote:
>
> > Quoting Johannes Schauer (2020-02-02 20:22:49)

> [...]

> > > 1. you can try the qcow2 image you built without autopkgtest by just running it
> > > inside qemu like so:
> > >
> > > $ qemu-system-x86_64 -enable-kvm -m 512 -serial unix:/tmp/ttyS0,server,nowait -drive "file=$HOME/Downloads/TEST/debian-unstable.qcow2,cache=unsafe,if=virtio,index=0"
> >
> > One non-obvious thing is missing here. After running the above command you have
> > a new unix socket under /tmp/ttyS0 that you should be able to connect to using
> > a serial terminal emulator. This is how autopkgtest connects to qemu. So after
> > running above command, check that everything works by running something like:
> >
> > $ minicom -D 'unix#/tmp/ttyS0'
>

> Hello and sorry for the long delay (I had been busy in too many things...).
>
> I tried the test you suggested and found out that the virtual machine
> fails to boot!
> The attached screenshot is what is shown in the graphical window...
> seemingly forever.
>
> I haven't even tried to connect to /tmp/ttyS0, as I guess a virtual
> machine that fails to boot will never bring the serial terminal
> emulator up and running...
>
> I am really ignorant about qemu: do you happen to have any idea on what
> went wrong?

here is a hunch:

Can you show me how you converted the tarball mmdebstrap produced into your
debian-unstable.qcow2 image?

Thanks!

cheers, josch

[Johannes Schauer]

Hi,

Quoting Francesco Poli (2020-02-19 23:21:55)

> On Tue, 18 Feb 2020 00:03:06 +0100 Johannes Schauer wrote:
> > Can you show me how you converted the tarball mmdebstrap
> > produced into your debian-unstable.qcow2 image?
>

> Sure, the command was:
>
> $ mmdebstrap-autopkgtest-qemu
>
> which is a small script I am trying to develop, based on your suggested
> command.
> It is still experimental, needs to be improved in many aspects, and has
> not yet produced any working QEMU image (!).
> I wanted to send it to you as a contribution to mmdebstrap, once it was
> mature enough.
> I am sending it to you now, as a preview (not to be included in the
> official package, yet!). Any suggestion for improvement is more than
> welcome...
>
> Please note that I am running it on an amd64 box. Hence:
>
> $ dpkg --print-architecture
> amd64
>
>
> I tried the image again, after adding my regular user to the 'kvm'
> group (I hadn't done so, out of ignorance about QEMU/KVM), but the
> result was the same:
>
> $ qemu-system-x86_64 -enable-kvm -m 512 -serial unix:/tmp/ttyS0,server,nowait -drive "file=./debian-unstable.qcow2,cache=unsafe,if=virtio,index=0"
>
> or
>
> $ qemu-system-x86_64 -m 512 -serial unix:/tmp/ttyS0,server,nowait -drive "file=./debian-unstable.qcow2,cache=unsafe,if=virtio,index=0"
>
> or
>
> $ qemu-system-x86_64 -enable-kvm -m 512 -serial unix:/tmp/ttyS0,server,nowait -drive "file=./debian-unstable.img,format=raw,cache=unsafe,if=virtio,index=0"
>
> always starts a virtual machine that hangs (seemingly) forever, at the
> "Booting from Hard Disk..." stage.
>
>
> I really cannot understand what's going on.

I have an idea. When I tried running guestfish on salsa CI it also would not
boot at all. The image that was created by guestfish just wasn't bootable for
some reason. After much tinkering, I added the following to the end of the
guestfish invocation:

: sync : umount / : shutdown

And that worked. Since there seem to be some machines where these additions to
guestfish are needed, I also now put these into all guestfish invocations and
examples in mmdebstrap but that version isn't released yet and you would have
had to look into the current git master.

Try it out and tell me if it helped!

Thanks!

cheers, josch

[Johannes Schauer]

Hi,

Quoting Francesco Poli (2020-02-21 23:14:12)
> I modified my script (see the current version attached to this message) and
> tried again.
>
> $ env | grep SOURCE
> SOURCE_DATE_EPOCH=1582320746
> $ mmdebstrap-autopkgtest-qemu
> [...]
> I: removing tempdir ${HOME}/Downloads/TEST/mmdebstrap.vt5NxWQb_6...
> libguestfs: error: /usr/bin/supermin exited with error status 1.
> To see full error messages you may need to enable debugging.
> Do:
> export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
> and run the command again. For further information, read:
> http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
> You can also run 'libguestfs-test-tool' and post the *complete* output
> into a bug report or message to the libguestfs mailing list.

did you try running guestfish with these env variables to figure out what's
wrong?

> The qcow2 image is suspiciously small:
>
> $ ls --si -l debian-unstable.*
> -rw-rw---- 1 $USER $GROUP 2.2G Feb 21 22:34 debian-unstable.img
> -rw-r----- 1 $USER $GROUP 197k Feb 21 22:34 debian-unstable.qcow2
> -rw-r--r-- 1 $USER $GROUP 629M Feb 21 22:34 debian-unstable.tar
>
> and actually fails to boot (but for different reasons, with respect to
> the previous attempt), see the attached screenshot.
> I tried both the qcow2 image and the raw image:

>
> $ qemu-system-x86_64 -enable-kvm -m 512 -serial unix:/tmp/ttyS0,server,nowait -drive "file=./debian-unstable.qcow2,cache=unsafe,if=virtio,index=0"

> $ qemu-system-x86_64 -enable-kvm -m 512 -serial unix:/tmp/ttyS0,server,nowait -drive "file=./debian-unstable.img,format=raw,cache=unsafe,if=virtio,index=0"
>

> with identical results: after attempting all possible boot devices
> (including a network boot!), it bails out with "No bootable device"
> error message.
>
> I also tried:
>
> $ export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
> $ SIZE="2GiB"
> $ guestfish --new debian-unstable_DEBUG.img=disk:"$SIZE" -- \

> part-disk /dev/sda mbr : \
> part-set-bootable /dev/sda 1 true : \
> mkfs ext2 /dev/sda1 : mount /dev/sda1 / : \
> tar-in debian-unstable.tar / : \
> extlinux / : \

> copy-in extlinux.conf / : \
> sync : umount / : shutdown
>
> which produced an unmanageable quantity of output strings, with an
> apparently endless stream line pairs identical to:
>
> guestfsd: receive_file: reading length word
> guestfsd: receive_file: got chunk: cancel = 0x0, len = 8192, buf = 0x560e1b866690
>
> until I hit [Ctrl+C]...
>
> Does it make any sense?

No, it doesn't I have never seen the effects you see.

I have more ideas but I think you first somehow have to fix the guestfish
problems that you are experiencing. What version of Debian and guestfish are
you running?

Thanks!

cheers, josch

[Johannes Schauer]

Hi,

Quoting Johannes Schauer (2020-02-22 08:14:55)

for what it's worth I'm attaching the script I am using to re-generate my
autopkgtest qemu image. I uploaded a few packages today and just successfully
used that script so I can confirm it working.

Thanks!

cheers, josch

[Francesco Poli]

On Sat, 22 Feb 2020 08:14:55 +0100 Johannes Schauer wrote:

[...]

> Quoting Francesco Poli (2020-02-21 23:14:12)

[...]

> > Does it make any sense?
>
> No, it doesn't I have never seen the effects you see.

As I have already said, I am apparently very unlucky with this effort
of mine!

>
> I have more ideas but I think you first somehow have to fix the guestfish
> problems that you are experiencing. What version of Debian and guestfish are
> you running?

I am running Debian testing (bullseye) and I was using
libguestfs-tools/1:1.40.2-7 (currently in testing).
When you asked, I noticed that there is a version rebuilt from the same
source in unstable, and I upgraded to it.

I have just retested my script with libguestfs-tools/1:1.40.2-7+b1:

$ apt policy libguestfs-tools
libguestfs-tools:
Installed: 1:1.40.2-7+b1
Candidate: 1:1.40.2-7+b1
Version table:
*** 1:1.40.2-7+b1 500
500 http://deb.debian.org/debian unstable/main amd64 Packages
100 /var/lib/dpkg/status
1:1.40.2-7 800
800 http://deb.debian.org/debian testing/main amd64 Packages

but the result was the same.
At the end of the script guestfish fails with the same error and
qemu-img produces a really tiny .qcow2 image, which does not seem to
contain any bootable partition.

The mystery deepens, as soon as I try to issue the guestfish command
manually (after the script exited):

$ guestfish --new debian-unstable_DEBUG.img=disk:"$SIZE" -- part-disk /dev/sda mbr : part-set-bootable /dev/sda 1 true : mkfs ext2 /dev/sda1 : mount /dev/sda1 / : tar-in debian-unstable.tar / : extlinux / : copy-in extlinux.conf / : sync : umount / : shutdown

which exits successfully and produces a .img file which qemu-img is
able to convert into a reasonably sized .qcow2 file:

$ ls --si -ltrF debian-unstable*
-rw-r--r-- 1 $USER $GROUP 629M Feb 22 18:30 debian-unstable.tar
-rw-rw---- 1 $USER $GROUP 2.2G Feb 22 18:30 debian-unstable.img
-rw-r----- 1 $USER $GROUP 197k Feb 22 18:30 debian-unstable.qcow2
-rw-rw---- 1 $USER $GROUP 2.2G Feb 22 18:31 debian-unstable_DEBUG.img
-rw-r----- 1 $USER $GROUP 707M Feb 22 18:31 debian-unstable_DEBUG.qcow2

Nevertheless, debian-unstable_DEBUG.qcow2 does not boot with:

$ qemu-system-x86_64 -enable-kvm -m 512 -serial unix:/tmp/ttyS0,server,nowait -drive "file=./debian-unstable_DEBUG.qcow2,cache=unsafe,if=virtio,index=0"

It hangs exactly like the first image I obtained weeks ago (that is to
say, at the "Booting from Hard Disk..." message.

I cannot understand how (what really appears to be) the same guestfish
command behaves differently inside the script and outside (after the
end of the script)!
This looks like black magic to me... :-|

[Francesco Poli]

On Sat, 22 Feb 2020 19:03:48 +0100 Johannes Schauer wrote:

[...]

> for what it's worth I'm attaching the script I am using to re-generate my
> autopkgtest qemu image. I uploaded a few packages today and just successfully
> used that script so I can confirm it working.

I cannot spot any significant difference between your script and mine
(apart from the fact that I have recently added "sync : umount / :
shutdown" to the guestfish command-line, as you suggested)...

I really cannot understand what's going on! :-(

[Johannes Schauer]

Hi,

did you get any further with this problem?

cheers, josch

Quoting Francesco Poli (2020-02-24 23:51:52)

[Francesco Poli]

On Tue, 05 May 2020 19:34:31 +0200 Johannes Schauer wrote:

> Hi,

Hello Johannes, glad to hear from you.

>
> did you get any further with this problem?

Unfortunately, I failed to progress any further.

I have just retried (who knows, after many system upgrades?!?).
Bad luck! Once again, I am stuck at the following error:

[...]
I: removing tempdir ${HOME}/Downloads/TEST/mmdebstrap.9R3P2pDSAZ...

libguestfs: error: /usr/bin/supermin exited with error status 1.
To see full error messages you may need to enable debugging.
Do:
export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
and run the command again. For further information, read:
http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
You can also run 'libguestfs-test-tool' and post the *complete* output
into a bug report or message to the libguestfs mailing list.

and I really do not know how to debug this.

That's unfortunate, since mmdebstrap looks very promising, but I seem
to be unable to make it work correctly... :-(

[Johannes Schauer]

Hi Francesco,

Quoting Francesco Poli (2020-05-06 19:40:37)

> > did you get any further with this problem?
>
> Unfortunately, I failed to progress any further.

I uploaded a new mmdebstrap version.

I am using the attached script to build my own autopkgtest qemu images and it
works fine for me. Maybe you want to try again?

Thanks!

cheers, josch

[Francesco Poli]

On Thu, 03 Sep 2020 22:53:13 +0200 Johannes Schauer wrote:

> Hi Francesco,

Hello Johannes, thanks for following up my bug report!

I have just retried with my script (which seems to do the same things as yours, except that it does set the unshare mode, since I have kernel.unprivileged_userns_clone = 0).

Unfortunately, I still see the same guestfish error:

I: automatically chosen mode: fakechroot

I: chroot architecture amd64 is equal to the host's architecture

I: automatically chosen format: tar
I: using ${HOME}/Downloads/TEST/mmdebstrap.Cd9NV53RA7 as tempdir
[...]
I: creating tarball...
I: done
I: removing tempdir ${HOME}/Downloads/TEST/mmdebstrap.Cd9NV53RA7...
I: success in 107.2860 seconds

libguestfs: error: /usr/bin/supermin exited with error status 1.
To see full error messages you may need to enable debugging.
Do:
export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
and run the command again. For further information, read:
http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
You can also run 'libguestfs-test-tool' and post the *complete* output
into a bug report or message to the libguestfs mailing list.

Once again, the .qcow2 image is tiny and the .img does not boot with

$ qemu-system-x86_64 -enable-kvm -m 512 \
-serial unix:/tmp/ttyS0,server,nowait -drive \
"file=./debian-unstable.img,format=raw,cache=unsafe,if=virtio,index=0"

After attempting all possible boot devices, including a network boot,

it bails out with "No bootable device" error message.

[Johannes Schauer]

Hi Francesco,

Quoting Francesco Poli (2020-09-04 18:18:32)

I have some more insights. After having gotten the mmdebstrap/guestfish thing
running on salsaci, debci and jenkins I also happened to see stuff that looked
very similar to the errors you are seeing. Going over my last emails to this
thread, the guestfish command I told you about was indeed missing something:
the installation of mbr.bin. So a better guestfish command is this one:

guestfish -N debian-unstable.img=disk:8G -- \
part-disk /dev/sda mbr : \

mkfs ext2 /dev/sda1 : \
mount /dev/sda1 / : \
tar-in debian-unstable.tar / : \

copy-in "extlinux.conf" / : \
upload /usr/lib/SYSLINUX/mbr.bin /mbr.bin : \
copy-file-to-device /mbr.bin /dev/sda size:440 : \
rm /mbr.bin : \
extlinux / : \
sync : \
umount / : \
part-set-bootable /dev/sda 1 true : \
shutdown

I have no idea why, but on my system, the disk boots even without writing
mbr.bin to the first 440 byte of the disk. Just to make sure, here is my
extlinux.conf:

default linux
timeout 0

label linux
kernel /vmlinuz
append initrd=/initrd.img root=/dev/vda1 rw net.ifnames=0 console=ttyS0

Also, you reported that the guestfish call fails for you and that if you set
LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1 then you get lots of lines like:

guestfsd: receive_file: reading length word
guestfsd: receive_file: got chunk: cancel = 0x0, len = 8192, buf = 0x560e1b866690

These lines come from the tar-in command. If it fails at that stage, maybe your
disk size is not large enough for your tarball and that's why it fails?

Thanks!

cheers, josch

[Francesco Poli]

On Wed, 11 Nov 2020 23:27:21 +0100 Johannes Schauer wrote:

> Hi Francesco,

Hello Johannes, thanks for the updated followup!

[...]

>So a better guestfish command is this one:
>
> guestfish -N debian-unstable.img=disk:8G -- \

[...]

> Just to make sure, here is my
> extlinux.conf:

[...]

> Also, you reported that the guestfish call fails

[...]

> maybe your
> disk size is not large enough for your tarball and that's why it fails?

I re-tried with the modified guestfish command, the modified
extlinux.conf file and requesting an 8 GiB size.

Unfortunately, the new test results in the following errors:


[...]
Setting up initramfs-tools-core (0.139) ...
Setting up initramfs-tools (0.139) ...
update-initramfs: deferring update (trigger activated)
Setting up linux-image-5.9.0-2-amd64 (5.9.6-1) ...
I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.9.0-2-amd64
I: /initrd.img.old is now a symlink to boot/initrd.img-5.9.0-2-amd64
I: /vmlinuz is now a symlink to boot/vmlinuz-5.9.0-2-amd64
I: /initrd.img is now a symlink to boot/initrd.img-5.9.0-2-amd64
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-5.9.0-2-amd64
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8125b-2.fw for module r8169
[...]
find: ‘/var/tmp/mkinitramfs_KOh7yF/lib/modules/5.9.0-2-amd64/kernel’: No such file or directory
cp: cannot stat '/usr/lib/klibc/bin/*': No such file or directory
cp: cannot stat '/lib/klibc-*.so': No such file or directory
E: /usr/share/initramfs-tools/hooks/klibc-utils failed with return 1.
update-initramfs: failed for /boot/initrd.img-5.9.0-2-amd64 with 1.
run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1
dpkg: error processing package linux-image-5.9.0-2-amd64 (--configure):
installed linux-image-5.9.0-2-amd64 package post-installation script subprocess returned error exit status 1
Setting up iptables (1.8.6-1) ...
[...]
dpkg: dependency problems prevent configuration of linux-image-amd64:
linux-image-amd64 depends on linux-image-5.9.0-2-amd64 (= 5.9.6-1); however:
Package linux-image-5.9.0-2-amd64 is not configured yet.

dpkg: error processing package linux-image-amd64 (--configure):
dependency problems - leaving unconfigured
[...]
Processing triggers for initramfs-tools (0.139) ...
Errors were encountered while processing:
linux-image-5.9.0-2-amd64
linux-image-amd64
E: Sub-process /usr/bin/dpkg returned an error code (1)
E: run_chroot failed: E: env --unset=APT_CONFIG --unset=TMPDIR /usr/sbin/chroot ${HOME}/Downloads/TEMP/mmdebstrap.HHRQ7rNuyL env --unset=APT_CONFIG --unset=TMPDIR apt-get --yes install -oAPT::Status-Fd=<$fd> -oDpkg::Use-Pty=false linux-image-amd64 adduser apt apt-utils bsdmainutils cpio cron debconf debconf-i18n debian-archive-keyring dmidecode e2fsprogs gcc-10-base gcc-9-base gpgv ifupdown init iproute2 iptables iputils-ping isc-dhcp-client isc-dhcp-common kmod less logrotate mawk nano netbase whiptail libpam-modules libpam-modules-bin libpam-runtime procps libreadline8 readline-common rsyslog sensible-utils passwd systemd systemd-sysv udev tasksel-data tzdata fdisk mount vim-common vim-tiny failed

W: listening on child socket failed:

I: removing tempdir ${HOME}/Downloads/TEMP/mmdebstrap.HHRQ7rNuyL...

libguestfs: error: /usr/bin/supermin exited with error status 1.
To see full error messages you may need to enable debugging.
Do:
export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
and run the command again. For further information, read:
http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
You can also run 'libguestfs-test-tool' and post the *complete* output
into a bug report or message to the libguestfs mailing list.

And the directory now includes the following files:

$ ls -altrF --si
total 218k
drwxrwx--- 3 $USER $USER 4.1k Nov 15 20:28 ../
-rw-r--r-- 1 $USER $USER 0 Nov 15 20:30 debian-unstable.tar
-rw-rw---- 1 $USER $USER 125 Nov 15 20:30 extlinux.conf
drwxr-xr-x 2 $USER $USER 4.1k Nov 15 20:30 .guestfs-1000/
-rw-rw---- 1 $USER $USER 8.6G Nov 15 20:30 debian-unstable.img
drwxrwx--- 3 $USER $USER 4.1k Nov 15 20:30 ./
-rw-r----- 1 $USER $USER 197k Nov 15 20:30 debian-unstable.qcow2


Any idea why initramfs-tools hooks are failing?

[Johannes Schauer]

Hi,

Quoting Francesco Poli (2020-11-15 20:43:49)

Which mode was mmdebstrap run in? Maybe this is something similar to this
initramfs-tools bug that popped up with fakechroot:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=944929

Can you share the whole script that you used for these results?

Thanks!

cheers, josch

[Johannes Schauer]

Hi,

Quoting Francesco Poli (2020-11-15 22:57:38)

> On Sun, 15 Nov 2020 21:05:08 +0100 Johannes Schauer wrote:
> > Which mode was mmdebstrap run in?
>

> $ mmdebstrap-autopkgtest-qemu 8GiB

> I: automatically chosen mode: fakechroot

> [...]

that's the culprit. Try using --mode=unshare. With fakechroot I see the errors
that you see. It works with unshare mode.

> > Maybe this is something similar to this initramfs-tools bug that popped up
> > with fakechroot:
> >
> > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=944929
>

> Isn't that fixed in unstable and in testing?

Yes, but the reason might be a similar one. Fakechroot works by preloading
libfakechroot.so and if any maintainer script messes with the environment
variables, then fakechroot gets disabled for that process.

> > Can you share the whole script that you used for these results?
>

> Sure, the modified script is attached.
>
> As I said before, the script is what I am trying to develop, based on
> your suggestions.

> It is still experimental, needs to be improved in many aspects, and has
> not yet produced any working QEMU image (!). I wanted to send it to you
> as a contribution to mmdebstrap, once it was mature enough.

> I am sending it to you as a preview (not to be included in the official
> package, yet!).

One suggestion would be to throw in a "set -e" at the beginning because
otherwise you'd have to check the exit status of each program execution
manually.

Thanks!

cheers, josch

[Francesco Poli]

On Sun, 15 Nov 2020 23:09:27 +0100 Johannes Schauer wrote:

[...]

> Quoting Francesco Poli (2020-11-15 22:57:38)

[...]

> > $ mmdebstrap-autopkgtest-qemu 8GiB
> > I: automatically chosen mode: fakechroot
> > [...]
>
> that's the culprit. Try using --mode=unshare. With fakechroot I see the errors
> that you see. It works with unshare mode.

As I have previously said, I am worried by security implications of
setting "kernel.unprivileged_userns_clone=1" with sysctl.
Bug #898446 is still being discussed...

[...]

> One suggestion would be to throw in a "set -e" at the beginning

Done, thanks for the suggestion!
It definitely makes sense for such a script.

[Johannes Schauer]

Hi,

Quoting Francesco Poli (2020-11-16 00:05:08)

> On Sun, 15 Nov 2020 23:09:27 +0100 Johannes Schauer wrote:
> > Quoting Francesco Poli (2020-11-15 22:57:38)

> As I have previously said, I am worried by security implications of
> setting "kernel.unprivileged_userns_clone=1" with sysctl.
> Bug #898446 is still being discussed...

I agree. Similarly I am worried about the security implications of running the
whole thing as root. It would be great if fakechroot would work. In the
meantime, I managed to track down the problem a bit. I put this shell snippet:

case "`FAKECHROOT_DETECT=1 /bin/echo`" in fakechroot*) echo LOADED;;*) echo NOT LOADED;;esac

Into various places like /etc/kernel/postinst.d/initramfs-tools,
/etc/kernel/postinst.d/initramfs-tools, /usr/sbin/update-initramfs,
/usr/sbin/mkinitramfs and /usr/share/initramfs-tools/hooks/klibc-utils and
found out that fakechroot still remains active all the way down to the deepest
level where the error is then produced by this line:

cp -pnL /usr/lib/klibc/bin/* "${DESTDIR}/bin"

The files in question *do* exist, what seems to be the problem are the
wildcards. For example this works:

mmdebstrap --mode=fakechroot --variant=apt --customize-hook='chroot "$1" sh -c "ls *"' unstable /dev/null

and so does this:

mmdebstrap --mode=fakechroot --variant=apt --customize-hook='chroot "$1" sh -c "ls ./*"' unstable /dev/null

But this fails:

mmdebstrap --mode=fakechroot --variant=apt --customize-hook='chroot "$1" sh -c "ls /*"' unstable /dev/null

So as soon as the wildcard is part of an absolute path, things start breaking.

Do you have the time to investigate further on this issue? This does not seem
to be a problem of initramfs-tools or a problem of fakechroot not being enabled
but a weird problem with fakechroot.

Thanks!

cheers, josch

[Francesco Poli]

On Mon, 16 Nov 2020 00:28:57 +0100 Johannes Schauer wrote:

[...]

> Do you have the time to investigate further on this issue?

As you can see, not much time, unfortunately! :-(
And I am sad about this, believe me.

> This does not seem
> to be a problem of initramfs-tools or a problem of fakechroot not being enabled
> but a weird problem with fakechroot.

I don't know whether I can get around to investigating the fakechroot
failure, but, in the meanwhile I noticed that #898446 has been closed
and a decision has been made.
It was apparently concluded that enabling
"kernel.unprivileged_userns_clone=1" by default is better than
disabling it by default.

As I have [previously said], I am not brave enough (and knowledgeable
enough on this topic) to diverge from the default Debian settings.

[previously said]: <https://bugs.debian.org/944485#26>

However, now the Debian default has changed:

$ /sbin/sysctl kernel.unprivileged_userns_clone
kernel.unprivileged_userns_clone = 1

As a consequence, I feel like giving the "unshare" mode of mmdebstrap a
try.

I tried again with my work-in-progress script:

$ cd ~/Downloads/
$ mmdebstrap-autopkgtest-qemu 8GiB
I: automatically chosen mode: unshare

I: chroot architecture amd64 is equal to the host's architecture
I: automatically chosen format: tar

I: using ${HOME}/Downloads/mmdebstrap.43LR3zoGLq as tempdir
E: cannot create ${HOME}/Downloads: Permission denied; cannot create ${HOME}/Downloads/mmdebstrap.43LR3zoGLq: Permission denied; cannot create ${HOME}/Downloads/mmdebstrap.43LR3zoGLq//etc: Permission denied; cannot create ${HOME}/Downloads/mmdebstrap.43LR3zoGLq//etc/apt: Permission denied; cannot create ${HOME}/Downloads/mmdebstrap.43LR3zoGLq//etc/apt/apt.conf.d: Permission denied

W: listening on child socket failed:

I: removing tempdir ${HOME}/Downloads/mmdebstrap.43LR3zoGLq...
E: unable to chdir() to parent directory of ${HOME}/Downloads/mmdebstrap.43LR3zoGLq: Permission denied
E: remove_tree failed


OK, this does not work at all.
I tried to read the relevant section in the mmdebstrap(1) man page, and
there's some recommendation about an --unshare-helper option.

But to be honest, I failed to understand what I am supposed to do with
this --unshare-helper option.
Could you please clarify?

Please remember that my script exports TMPDIR='.', in order to avoid
using a tightly sized /tmp partition, where several GB of data would
definitely not fit.

Thanks for your time and patience.

[Johannes Schauer Marin Rodrigues]

Quoting Francesco Poli (2021-03-27 16:18:26)

> As a consequence, I feel like giving the "unshare" mode of mmdebstrap a
> try.
>
> I tried again with my work-in-progress script:
>
> $ cd ~/Downloads/
> $ mmdebstrap-autopkgtest-qemu 8GiB
> I: automatically chosen mode: unshare
> I: chroot architecture amd64 is equal to the host's architecture
> I: automatically chosen format: tar
> I: using ${HOME}/Downloads/mmdebstrap.43LR3zoGLq as tempdir
> E: cannot create ${HOME}/Downloads: Permission denied; cannot create ${HOME}/Downloads/mmdebstrap.43LR3zoGLq: Permission denied; cannot create ${HOME}/Downloads/mmdebstrap.43LR3zoGLq//etc: Permission denied; cannot create ${HOME}/Downloads/mmdebstrap.43LR3zoGLq//etc/apt: Permission denied; cannot create ${HOME}/Downloads/mmdebstrap.43LR3zoGLq//etc/apt/apt.conf.d: Permission denied
> W: listening on child socket failed:
> I: removing tempdir ${HOME}/Downloads/mmdebstrap.43LR3zoGLq...
> E: unable to chdir() to parent directory of ${HOME}/Downloads/mmdebstrap.43LR3zoGLq: Permission denied
> E: remove_tree failed
>
>
> OK, this does not work at all.
> I tried to read the relevant section in the mmdebstrap(1) man page, and
> there's some recommendation about an --unshare-helper option.
>
> But to be honest, I failed to understand what I am supposed to do with
> this --unshare-helper option.
> Could you please clarify?

Quoting from the man page:

"For correct ownership information, the directory must be accessed from a
user namespace with the right subuid/subgid offset"

Which part is unclear about that?

> Please remember that my script exports TMPDIR='.', in order to avoid
> using a tightly sized /tmp partition, where several GB of data would
> definitely not fit.

I also don't understand which part of the error message above is unclear. It
tells you that you don't have permission to create the directories which means
that whatever your '${HOME}/Downloads' is (or one of its parent directories),
has permissions set such that the directory cannot be created and/or accessed.

Thanks!

cheers, josch

[Francesco Poli]

On Sat, 27 Mar 2021 20:15:29 +0100 Johannes Schauer Marin Rodrigues
wrote:

> Quoting Francesco Poli (2021-03-27 16:18:26)

[...]

> > I tried to read the relevant section in the mmdebstrap(1) man page, and
> > there's some recommendation about an --unshare-helper option.
> >
> > But to be honest, I failed to understand what I am supposed to do with
> > this --unshare-helper option.
> > Could you please clarify?
>
> Quoting from the man page:
>
> "For correct ownership information, the directory must be accessed from a
> user namespace with the right subuid/subgid offset"
>
> Which part is unclear about that?

Thanks for your prompt reply!

Well, the problem is probably my total ignorance about unshare, user
namespaces, and so forth...

First of all, I do not know what exactly subuid and subgid are.

I suppose they have something to do with the uid and gid seen from
within the namespace (I noticed that the temporary directory was
created with high values of uid and gid, not corresponding to any
existing user or group).
I tried to take a look at some man pages (unshare(1),
user_namespaces(7), ...) in order to understand something more, but
there seems to be much more information than the basics, and I failed
to pinpoint where I can read the bare minimum I need to know...
My bad, of course!
But anyway, if you know a good reference to a short explanation of the
topic (ideally a man page), I would suggest to cite it in the
mmdebstrap man page.

After that, I can guess that the problem is that, inside the unshared
namespace, there's no permission to create/write files in my
~/Downloads directory.
But what is not clear to me is what I am supposed to do with this
problem.

What would you suggest?

[Johannes Schauer Marin Rodrigues]

Quoting Francesco Poli (2021-03-28 20:24:12)

> Well, the problem is probably my total ignorance about unshare, user
> namespaces, and so forth...
>
> First of all, I do not know what exactly subuid and subgid are.

man subuid? ;)

> I suppose they have something to do with the uid and gid seen from
> within the namespace (I noticed that the temporary directory was
> created with high values of uid and gid, not corresponding to any
> existing user or group).

Correct. So if some directory on the path of your $TMPDIR is not accessible by
that user with the very high uid/gid, then it obviously cannot create the
chroot directory.

> I tried to take a look at some man pages (unshare(1),
> user_namespaces(7), ...) in order to understand something more, but
> there seems to be much more information than the basics, and I failed
> to pinpoint where I can read the bare minimum I need to know...
> My bad, of course!
> But anyway, if you know a good reference to a short explanation of the
> topic (ideally a man page), I would suggest to cite it in the
> mmdebstrap man page.

Unfortunately, I do not know of such a reference. All I learned to add unshare
support to mmdebstrap was from reading source code of other tools. :(

> After that, I can guess that the problem is that, inside the unshared
> namespace, there's no permission to create/write files in my
> ~/Downloads directory.

Or one of its parent. You also need more than write access. You also need read
access. And directories need to be executable.

> But what is not clear to me is what I am supposed to do with this
> problem.
>
> What would you suggest?

Your whole problem started because you decided to deviate from the default and
chose a different TMPDIR. If you use custom options, then you should know what
you are doing. If you don't set TMPDIR, then it will work. If you do set
TMPDIR, then you have to make sure that whatever path TMPDIR points to is setup
in a way that the unshared user is able to access it and write stuff into it.

Thanks!

cheers, josch

[Francesco Poli]

On Sun, 28 Mar 2021 21:04:24 +0200 Johannes Schauer Marin Rodrigues wrote:

> Quoting Francesco Poli (2021-03-28 20:24:12)
> > Well, the problem is probably my total ignorance about unshare, user
> > namespaces, and so forth...
> >
> > First of all, I do not know what exactly subuid and subgid are.
>
> man subuid? ;)

OK, I have read subuid(5) man page.
Now I know which subordinate UIDs my user can use within a namespace.

But do I understand correctly that I cannot predict which one will be
used next time I try to start mmdebstrap in unshare mode?

>
> > I suppose they have something to do with the uid and gid seen from
> > within the namespace (I noticed that the temporary directory was
> > created with high values of uid and gid, not corresponding to any
> > existing user or group).
>
> Correct. So if some directory on the path of your $TMPDIR is not accessible by
> that user with the very high uid/gid, then it obviously cannot create the
> chroot directory.

This seems to imply that I cannot use anything within my home directory
as $TMPDIR ...
Is this the case?

If this is indeed the case, and /tmp is too tightly sized (SSD
partition of size about 870 MB), where can I put my $TMPDIR ?

/dev/shm seems to be risky (its size is about 8.3 GB, less than half
the system memory).

Suggestions?

I am assuming that creating a QEMU/KVM image of size, say, 200 MiB is
not enough as an autopkgtest-virt-qemu testbed.
I was aiming at 4 GiB or 8 GiB, but maybe I am off-track here.
Please advise!

>
> > I tried to take a look at some man pages (unshare(1),
> > user_namespaces(7), ...) in order to understand something more, but
> > there seems to be much more information than the basics, and I failed
> > to pinpoint where I can read the bare minimum I need to know...
> > My bad, of course!
> > But anyway, if you know a good reference to a short explanation of the
> > topic (ideally a man page), I would suggest to cite it in the
> > mmdebstrap man page.
>
> Unfortunately, I do not know of such a reference. All I learned to add unshare
> support to mmdebstrap was from reading source code of other tools. :(

I see, the good old "Use the Force, read the Source!". ;-)

>
> > After that, I can guess that the problem is that, inside the unshared
> > namespace, there's no permission to create/write files in my
> > ~/Downloads directory.
>
> Or one of its parent. You also need more than write access. You also need read
> access. And directories need to be executable.
>
> > But what is not clear to me is what I am supposed to do with this
> > problem.
> >
> > What would you suggest?
>
> Your whole problem started because you decided to deviate from the default and
> chose a different TMPDIR. If you use custom options, then you should know what
> you are doing. If you don't set TMPDIR, then it will work. If you do set
> TMPDIR, then you have to make sure that whatever path TMPDIR points to is setup
> in a way that the unshared user is able to access it and write stuff into it.

As said above: I chose a different TMPDIR, since /tmp turned out to not
be big enough.
I am open to suggestions on how to work around this issue.

And by the way, I still do not understand how the --unshare-helper can
help me... as long as it can help me at all!
Could you please clarify?

Thanks for your time and great patience. :-)

[Johannes Schauer Marin Rodrigues]

Quoting Francesco Poli (2021-03-29 23:26:27)

> On Sun, 28 Mar 2021 21:04:24 +0200 Johannes Schauer Marin Rodrigues wrote:
>
> > Quoting Francesco Poli (2021-03-28 20:24:12)
> > > Well, the problem is probably my total ignorance about unshare, user
> > > namespaces, and so forth...
> > >
> > > First of all, I do not know what exactly subuid and subgid are.
> >
> > man subuid? ;)
>
> OK, I have read subuid(5) man page.
> Now I know which subordinate UIDs my user can use within a namespace.
>
> But do I understand correctly that I cannot predict which one will be
> used next time I try to start mmdebstrap in unshare mode?

Theoretically your user can indeed use any uid from that range for its purposes
but in practice I never saw a program choose the uid mapping arbitrarily.
Instead (and that's also what mmdebstrap does) it will choose the first subuid
that you can use for the root user (uid 0) and then all the remaining uids from
then on. So my /etc/subuid says:

josch:100000:65536

So my root user in unshare mode will always get the uid 100000 as seen from the
outside and a user with uid 1000 inside the unshare environment will have uid
101000 as seen from the outside.

This is what happens when you use tools like lxc-usernsexec without the -m
option which allows you to use a custom range. Other programs require you to
know your subuid offset from /etc/subuid and you have to provide it manually
(see systemd-nspawn in the mmdebstrap man page).

But I don't think that this kind of information belongs into the mmdebstrap man
page.

> > > I suppose they have something to do with the uid and gid seen from within
> > > the namespace (I noticed that the temporary directory was created with
> > > high values of uid and gid, not corresponding to any existing user or
> > > group).
> >
> > Correct. So if some directory on the path of your $TMPDIR is not accessible by
> > that user with the very high uid/gid, then it obviously cannot create the
> > chroot directory.
>
> This seems to imply that I cannot use anything within my home directory
> as $TMPDIR ...
> Is this the case?

Not necessarily. If your $HOME is readable by "everybody" and if you create a
directory inside your $HOME which allows "everybody" to create stuff in it then
you can use that as your $TMPDIR. Best even if you chmod that directory with
the sticky bit in the same way as it's done for /tmp. If the directory is not
directly inside your $HOME, then all directories in the path must be world
readable. Only the last component must be world writable.

I think the "world writable" assumption is a fair one for a temporary
directory.

> If this is indeed the case, and /tmp is too tightly sized (SSD partition of
> size about 870 MB), where can I put my $TMPDIR ?

How many chroots do you want to put into your /tmp?

$ mmdebstrap unstable | wc -c
...
196625920

That's 188 MB.

> /dev/shm seems to be risky (its size is about 8.3 GB, less than half the
> system memory).
>
> Suggestions?
>
> I am assuming that creating a QEMU/KVM image of size, say, 200 MiB is
> not enough as an autopkgtest-virt-qemu testbed.
> I was aiming at 4 GiB or 8 GiB, but maybe I am off-track here.
> Please advise!

That depends on what you want to do inside that qemu environment. And it's
totally possible to create a 10 GB disk image in your $HOME while only using
200 MB in your /tmp for the chroot.

> > > But what is not clear to me is what I am supposed to do with this
> > > problem.
> > >
> > > What would you suggest?
> >
> > Your whole problem started because you decided to deviate from the default and
> > chose a different TMPDIR. If you use custom options, then you should know what
> > you are doing. If you don't set TMPDIR, then it will work. If you do set
> > TMPDIR, then you have to make sure that whatever path TMPDIR points to is setup
> > in a way that the unshared user is able to access it and write stuff into it.
>
> As said above: I chose a different TMPDIR, since /tmp turned out to not
> be big enough.
> I am open to suggestions on how to work around this issue.
>
> And by the way, I still do not understand how the --unshare-helper can
> help me... as long as it can help me at all!
> Could you please clarify?

It cannot help you. The option is undocumented because its interface is
unstable. What the --unshare-helper option does is to provide you similar
functionality to lxc-usernsexec -- lxc-unshare -s 'MOUNT|PID|UTSNAME|IPC'
without installing the lxc package.

> Thanks for your time and great patience. :-)

You are welcome. I would love to hear things I could add to the mmdebstrap man
page. I fear that an introduction to Linux user namespaces would not be
fitting.

[Francesco Poli]

On Tue, 30 Mar 2021 02:04:03 +0200 Johannes Schauer Marin Rodrigues wrote:

[...]

> That depends on what you want to do inside that qemu environment. And it's
> totally possible to create a 10 GB disk image in your $HOME while only using
> 200 MB in your /tmp for the chroot.

Well... I tried dropping the TMPDIR setting and exporting in the script.
But:

$ cd ~/Downloads/
$ mmdebstrap-autopkgtest-qemu 8GiB
I: automatically chosen mode: unshare
I: chroot architecture amd64 is equal to the host's architecture
I: automatically chosen format: tar

I: using /tmp/mmdebstrap.h6DOq1LzM2 as tempdir
[...]
Setting up linux-image-5.10.0-5-amd64 (5.10.26-1) ...
[...]
cp: error writing '/var/tmp/mkinitramfs_chjGze//usr/lib/modules/5.10.0-5-amd64/kernel/drivers/input/mouse/psmouse.ko': No space left on device

E: Sub-process /usr/bin/dpkg returned an error code (1)

E: run_chroot failed: E: env --unset=APT_CONFIG --unset=TMPDIR /usr/sbin/chroot /tmp/mmdebstrap.h6DOq1LzM2 apt-get --yes install -oAPT::Status-Fd=<$fd> -oDpkg::Use-Pty=false dmidecode linux-image-amd64 logrotate sensible-utils fdisk rsyslog systemd libpam-modules-bin iproute2 vim-common passwd iputils-ping init e2fsprogs kmod adduser tzdata ifupdown udev vim-tiny libreadline8 gcc-10-base mawk apt apt-utils gpgv gcc-9-base mount libpam-runtime whiptail debian-archive-keyring procps debconf-i18n tasksel-data readline-common nano netbase isc-dhcp-client systemd-sysv nftables cpio debconf less isc-dhcp-common libpam-modules cron failed

W: listening on child socket failed:

I: removing tempdir /tmp/mmdebstrap.h6DOq1LzM2...


The same happens with:

$ mmdebstrap-autopkgtest-qemu 1GiB


I cannot understand why you needed less than 200 MB ...

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Francesco Poli (2022-01-06 19:22:28)
> On Wed, 31 Mar 2021 22:45:02 +0200 Francesco Poli wrote:
>
> [...]

> > I cannot understand why you needed less than 200 MB ...
>

> Hello again Johannes,
> I decided to try again (with mmdebstrap/0.8.2-1 and the attached
> work-in-progress script), but I obtained the following error, which is
> completely new to me:

>
> $ cd ~/Downloads/
> $ mmdebstrap-autopkgtest-qemu 8GiB
> I: automatically chosen mode: unshare
> I: chroot architecture amd64 is equal to the host's architecture
> I: automatically chosen format: tar

> I: using /tmp/mmdebstrap.YCxe0VQ5LQ as tempdir
> dpkg: warning: failed to open configuration file '${HOME}/.dpkg.cfg' for reading: Permission denied
> I: running apt-get update...
> done
> I: downloading packages with apt...
> done
> E: nothing got downloaded
> W: listening on child socket failed: Undefined subroutine &Devel::Cover::get_coverage called at /usr/bin/mmdebstrap line 105.
>
> I: removing tempdir /tmp/mmdebstrap.YCxe0VQ5LQ...
>
> And the resulting tar archive is empty:
>
> $ ls -altrF --si
> total 17k
> drwxrwx--- 2 $USER $USER 4.1k Jan 6 19:10 ./
> -rw-r--r-- 1 $USER $USER 0 Jan 6 19:10 debian-unstable.tar
> drwxr-x--- 113 $USER $USER 13k Jan 6 19:12 ../
>
>
> What's going on?

that was #1003191. Please try again once 0.8.3-1 hits your mirror.

Thanks!

cheers, josch

[Francesco Poli]

On Sat, 08 Jan 2022 14:36:35 +0100 Johannes Schauer Marin Rodrigues wrote:

[...]

> Please try again once 0.8.3-1 hits your mirror.

I have just tried:

$ apt policy mmdebstrap
mmdebstrap:
Installed: 0.8.3-1
Candidate: 0.8.3-1
Version table:
*** 0.8.3-1 800

800 http://deb.debian.org/debian testing/main amd64 Packages

500 http://deb.debian.org/debian unstable/main amd64 Packages
100 /var/lib/dpkg/status

$ mmdebstrap-autopkgtest-qemu 8GiB
I: automatically chosen mode: unshare
I: chroot architecture amd64 is equal to the host's architecture
I: automatically chosen format: tar

I: using /tmp/mmdebstrap.5tbbK6Pj4J as tempdir
[...]
Unpacking linux-image-5.15.0-2-amd64 (5.15.5-2) ...
dpkg: error processing archive /tmp/apt-dpkg-install-GmBlUJ/69-linux-image-5.15.0-2-amd64_5.15.5-2_amd64.deb (--unpack):
cannot copy extracted data for './lib/modules/5.15.0-2-amd64/kernel/sound/usb/snd-usb-audio.ko' to '/lib/modules/5.15.0-2-amd64/kernel/sound/usb/snd-usb-audio.ko.dpkg-new': failed to write (No space left on device)
dpkg-deb: error: paste subprocess was killed by signal (Broken pipe)
[...]
E: Sub-process env returned an error code (1)
E: run_chroot failed: E: apt-get -o Dir::Bin::dpkg=env -o DPkg::Options::=--unset=TMPDIR -o DPkg::Options::=dpkg -o DPkg::Chroot-Directory=/tmp/mmdebstrap.5tbbK6Pj4J --yes install -oAPT::Status-Fd=<$fd> -oDpkg::Use-Pty=false ?narrow(?or(?archive(^sid$),?codename(^sid$)),?architecture(amd64),?or(?priority(required),?priority(important))) linux-image-amd64 failed

W: listening on child socket failed:

I: removing tempdir /tmp/mmdebstrap.5tbbK6Pj4J...
E: mmdebstrap failed to run


So I am back to the previous situation.
My /tmp is not big enough for the needs of mmdebstrap:

$ df --si /tmp
Filesystem Size Used Avail Use% Mounted on
/dev/md3 869M 930k 806M 1% /tmp

Have you had a chance to investigate why you need less than 200 MB,
while I need more than 869 MB?

Any idea how to solve this issue?

[Johannes Schauer Marin Rodrigues]

Quoting Francesco Poli (2022-01-10 23:12:34)

You could set TMPDIR to a location that has enough space.

[Francesco Poli]

On Mon, 10 Jan 2022 23:18:36 +0100 Johannes Schauer Marin Rodrigues
wrote:

[...]

> You could set TMPDIR to a location that has enough space.

I had tried, if you recall.
But you told me that the temporary directory must be world-writable
(and preferably with the sticky bit set) and all the directories in the
path must be world-readable.

This rules out anything within my home directory (setting my home
directory as world-readable is out of the question).

We have already seen that my /tmp lives in a partition which is too
small.

I would rather not create some non-FHS directory for this specific
purpose: it would effectively be a second non-standard /tmp directory
somewhere...

So, what's left?!?

Is /dev/shm a possible choice?

$ df --si /dev/shm

Filesystem Size Used Avail Use% Mounted on

tmpfs 8.3G 108M 8.2G 2% /dev/shm

It's half the size of my main memory and it's within the main memory.
Should I try to set TMPDIR=/dev/shm ?

[Johannes Schauer Marin Rodrigues]

Quoting Francesco Poli (2022-01-13 23:52:02)

> On Mon, 10 Jan 2022 23:18:36 +0100 Johannes Schauer Marin Rodrigues
> wrote:
>
> [...]
> > You could set TMPDIR to a location that has enough space.
>
> I had tried, if you recall.
> But you told me that the temporary directory must be world-writable
> (and preferably with the sticky bit set) and all the directories in the
> path must be world-readable.
>
> This rules out anything within my home directory (setting my home
> directory as world-readable is out of the question).
>
> We have already seen that my /tmp lives in a partition which is too
> small.
>
> I would rather not create some non-FHS directory for this specific
> purpose: it would effectively be a second non-standard /tmp directory
> somewhere...
>
> So, what's left?!?
>
> Is /dev/shm a possible choice?
>
> $ df --si /dev/shm
> Filesystem Size Used Avail Use% Mounted on
> tmpfs 8.3G 108M 8.2G 2% /dev/shm
>
> It's half the size of my main memory and it's within the main memory.
> Should I try to set TMPDIR=/dev/shm ?

yes, I think that's worth a try.

Other than that, you could run mmdebstrap with --mode=fakechroot which, in
comparison to unshare mode, doesn't require the world-readable path for the
TMPDIR.

Thanks!

cheers, josch

[Francesco Poli]

On Thu, 13 Jan 2022 23:59:49 +0100 Johannes Schauer Marin Rodrigues wrote:

> Quoting Francesco Poli (2022-01-13 23:52:02)
> > On Mon, 10 Jan 2022 23:18:36 +0100 Johannes Schauer Marin Rodrigues
> > wrote:
> >
> > [...]
> > > You could set TMPDIR to a location that has enough space.

[...]

> > Should I try to set TMPDIR=/dev/shm ?
>
> yes, I think that's worth a try.
>
> Other than that, you could run mmdebstrap with --mode=fakechroot which, in
> comparison to unshare mode, doesn't require the world-readable path for the
> TMPDIR.

I tried again with --mode=fakechroot and TMPDIR='.' (within my home
directory).

I again obtain the following error (as happened in the past, before I
started trying with the unshare mode):

[...]
I: creating tarball...
I: done

I: removing tempdir ${HOME}/Downloads/mmdebstrap.1LUIt_4tx7...
I: success in 132.1641 seconds

libguestfs: error: /usr/bin/supermin exited with error status 1.
To see full error messages you may need to enable debugging.
Do:
export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
and run the command again. For further information, read:
http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
You can also run 'libguestfs-test-tool' and post the *complete* output
into a bug report or message to the libguestfs mailing list.

and the resulting image fails to boot ("No bootable device."),
if I test it with the following command:

$ qemu-system-x86_64 -enable-kvm -m 512 -serial unix:/tmp/ttyS0,server,nowait -drive "file=./debian-unstable.img,format=raw,cache=unsafe,if=virtio,index=0"

I will (hopefully soon) try with TMPDIR=/dev/shm
and let you know.

[Francesco Poli]

On Wed, 23 Feb 2022 00:03:01 +0100 Francesco Poli wrote:

[...]

> I will (hopefully soon) try with TMPDIR=/dev/shm
> and let you know.

Unfortunately, it does not seem to work this way, either.

I: automatically chosen mode: unshare
I: chroot architecture amd64 is equal to the host's architecture
I: automatically chosen format: tar

I: using /dev/shm/mmdebstrap.kgUCDAC2QZ as tempdir
[...]
I: cleaning package lists and apt cache...
done
done

I: creating tarball...
I: done

I: removing tempdir /dev/shm/mmdebstrap.kgUCDAC2QZ...
I: success in 82.0568 seconds
libguestfs: error: open: usr/lib/SYSLINUX/mbr.bin: No such file or directory

and the resulting 'debian-unstable.img' image (size: 8.0 GiB == 8.6 GB)
fails to boot with ("Booting from Hard Disk..." which never completes),

if I test it with the following command:

$ qemu-system-x86_64 -enable-kvm -m 512 -serial unix:/tmp/ttyS0,server,nowait -drive "file=./debian-unstable.img,format=raw,cache=unsafe,if=virtio,index=0"

[Johannes Schauer Marin Rodrigues]

Quoting Francesco Poli (2022-02-24 23:14:43)

> On Wed, 23 Feb 2022 00:03:01 +0100 Francesco Poli wrote:
>
> [...]
> > I will (hopefully soon) try with TMPDIR=/dev/shm
> > and let you know.
>
> Unfortunately, it does not seem to work this way, either.
>
> I: automatically chosen mode: unshare
> I: chroot architecture amd64 is equal to the host's architecture
> I: automatically chosen format: tar
> I: using /dev/shm/mmdebstrap.kgUCDAC2QZ as tempdir
> [...]
> I: cleaning package lists and apt cache...
> done
> done
> I: creating tarball...
> I: done
> I: removing tempdir /dev/shm/mmdebstrap.kgUCDAC2QZ...
> I: success in 82.0568 seconds
> libguestfs: error: open: usr/lib/SYSLINUX/mbr.bin: No such file or directory
>
> and the resulting 'debian-unstable.img' image (size: 8.0 GiB == 8.6 GB)
> fails to boot with ("Booting from Hard Disk..." which never completes),
> if I test it with the following command:
>
> $ qemu-system-x86_64 -enable-kvm -m 512 -serial unix:/tmp/ttyS0,server,nowait -drive "file=./debian-unstable.img,format=raw,cache=unsafe,if=virtio,index=0"

Yes, it fails because you don't have /usr/lib/SYSLINUX/mbr.bin on your system.
Did you install the syslinux package? Sylinux/extlinux is the bootloader, so it
makes sense that the result doesn't boot after this error message.

Also notice, that the instructions of how to run all of this were recently
streamlined by another user, so you might want to double-check that your
scripts do the same thing:

https://gitlab.mister-muffin.de/josch/mmdebstrap/src/branch/main/mmdebstrap#L6844

If you don't have any luck with extlinux, then I recently figured out a way to
boot with grub2:

$ mmdebstrap --include=linux-image-amd64,grub2,systemd-sysv unstable rootfs.tar
$ guestfish -N debian-unstable.img=disk:2G -- \
part-disk /dev/sda mbr : \
part-set-bootable /dev/sda 1 true : \
mkfs ext2 /dev/sda1 : \
set-label /dev/sda1 rootfs : \
mount /dev/sda1 / : \
tar-in rootfs.tar / xattrs:true : \
command "grub-install /dev/sda" : \
command update-grub : \
sync : umount / : shutdown
$ qemu-system-x86_64 -m 512M -enable-kvm debian-unstable.img

This example is missing all the autopkgtest related bits but it's bootable.

Thanks!

cheers, josch

[Francesco Poli]

On Sat, 26 Feb 2022 00:35:01 +0100 Francesco Poli wrote:

[...]

> And it generates a 'debian-unstable.qcow2' image that can boot
[...]

However, it leaves a lot of "production scrap" within /dev/shm :

$ du --si -s /dev/shm/.guestfs-1000/
755M /dev/shm/.guestfs-1000/

More than 750 MB of "production scrap"!

Why does guestfish do this?

[Johannes Schauer Marin Rodrigues]

Hi Francesco,

Quoting Francesco Poli (2022-02-26 00:35:01)
> At least, it completes without any error.
> And it generates a 'debian-unstable.qcow2' image that can boot, if I
> test it with:
>
> $ qemu-system-x86_64 -enable-kvm -m 512 -serial unix:/tmp/ttyS0,server,nowait -drive "file=./debian-unstable.qcow2,cache=unsafe,if=virtio,index=0"
>
> I can even login as root (password-less!) in the graphical window that
> QEMU shows!
>
> Now, I remember that you said I should test the serial console with
> minicom on the /tmp/ttyS0 socket.
> I typically use picocom for serial console connections.
> Is there a way to make picocom connect to a UNIX socket?
>
> Maybe I should try after using socat, as suggested in this [howto].
>
> [howto]: <https://www.schmut.com/cheat-sheets/unix-serial-howto>
>
> Or is there a better way?

I don't know about picocom. Either of the following should work:

$ minicom -D unix#/tmp/ttyS0
$ nc -U /tmp/ttyS0
$ socat - UNIX-CONNECT:/tmp/ttyS0

> However, it leaves a lot of "production scrap" within /dev/shm :
>
> $ du --si -s /dev/shm/.guestfs-1000/
> 755M /dev/shm/.guestfs-1000/
>
> More than 750 MB of "production scrap"!
>
> Why does guestfish do this?

I don't know and I don't have these files in my own $TMPDIR.

In any case, if it works for you now, maybe you can close this bug?

Thanks!

cheers, josch

[Francesco Poli]

On Mon, 28 Feb 2022 11:11:48 +0100 Johannes Schauer Marin Rodrigues wrote:

> Hi Francesco,
>
> Quoting Francesco Poli (2022-02-26 00:35:01)

[...]

> > I typically use picocom for serial console connections.
> > Is there a way to make picocom connect to a UNIX socket?
> >
> > Maybe I should try after using socat, as suggested in this [howto].
> >
> > [howto]: <https://www.schmut.com/cheat-sheets/unix-serial-howto>
> >
> > Or is there a better way?
>
> I don't know about picocom. Either of the following should work:
>
> $ minicom -D unix#/tmp/ttyS0
> $ nc -U /tmp/ttyS0
> $ socat - UNIX-CONNECT:/tmp/ttyS0

I tried the last one (with socat) and I could login as root.

>
> > However, it leaves a lot of "production scrap" within /dev/shm :
> >
> > $ du --si -s /dev/shm/.guestfs-1000/
> > 755M /dev/shm/.guestfs-1000/
> >
> > More than 750 MB of "production scrap"!
> >
> > Why does guestfish do this?
>
> I don't know and I don't have these files in my own $TMPDIR.

I added a

rm -Rf ${TMPDIR}/.guestfs-*/

at the end of the script...

>
> In any case, if it works for you now, maybe you can close this bug?

Well, I first need to check that it actually works as a testbed for
autopkgtest (this was the initial goal of this incredible journey, if
you recall!).


Moreover, I tried to modify the script to more closely follow the
[streamlined suggestions].

[streamlined suggestions]: <https://gitlab.mister-muffin.de/josch/mmdebstrap/src/branch/main/mmdebstrap#L6844>

And, after doing so, the image was no longer able to boot!
It hanged (seemingly forever) while booting from hard disk.

I performed various tests (basically bisecting between the working
script and the failing one).
I found out that the issue was in the suggested 'extlinux.conf':

$ diff -u bin/BAD_mmdebstrap-autopkgtest-qemu bin/mmdebstrap-autopkgtest-qemu
--- bin/BAD_mmdebstrap-autopkgtest-qemu 2022-03-01 00:05:16.418739282 +0100
+++ bin/mmdebstrap-autopkgtest-qemu 2022-03-01 00:27:14.874469123 +0100
@@ -134,7 +134,7 @@

label linux
kernel /vmlinuz
-append initrd=/initrd.img root=/dev/vda1 console=ttyS0
+append initrd=/initrd.img root=/dev/vda1 rw net.ifnames=0 console=ttyS0
EOF

$GUESTFISH --new debian-unstable.img=disk:"$SIZE" -- \


I don't understand why, but this is how I fixed the script.
Should the documentation be fixed?

[Johannes Schauer Marin Rodrigues]

Quoting Francesco Poli (2022-03-01 00:49:50)

> > > However, it leaves a lot of "production scrap" within /dev/shm :
> > >
> > > $ du --si -s /dev/shm/.guestfs-1000/
> > > 755M /dev/shm/.guestfs-1000/
> > >
> > > More than 750 MB of "production scrap"!
> > >
> > > Why does guestfish do this?
> >
> > I don't know and I don't have these files in my own $TMPDIR.
>
> I added a
>
> rm -Rf ${TMPDIR}/.guestfs-*/
>
> at the end of the script...

Interesting. I don't see a directory like that anywhere on my system. Not even
while guestfish is running.

> > In any case, if it works for you now, maybe you can close this bug?
>
> Well, I first need to check that it actually works as a testbed for
> autopkgtest (this was the initial goal of this incredible journey, if
> you recall!).

Yes. :)

> Moreover, I tried to modify the script to more closely follow the
> [streamlined suggestions].
>
> [streamlined suggestions]: <https://gitlab.mister-muffin.de/josch/mmdebstrap/src/branch/main/mmdebstrap#L6844>
>
> And, after doing so, the image was no longer able to boot!
> It hanged (seemingly forever) while booting from hard disk.
>
> I performed various tests (basically bisecting between the working
> script and the failing one).
> I found out that the issue was in the suggested 'extlinux.conf':
>
> $ diff -u bin/BAD_mmdebstrap-autopkgtest-qemu bin/mmdebstrap-autopkgtest-qemu
> --- bin/BAD_mmdebstrap-autopkgtest-qemu 2022-03-01 00:05:16.418739282 +0100
> +++ bin/mmdebstrap-autopkgtest-qemu 2022-03-01 00:27:14.874469123 +0100
> @@ -134,7 +134,7 @@
>
> label linux
> kernel /vmlinuz
> -append initrd=/initrd.img root=/dev/vda1 console=ttyS0
> +append initrd=/initrd.img root=/dev/vda1 rw net.ifnames=0 console=ttyS0
> EOF
>
> $GUESTFISH --new debian-unstable.img=disk:"$SIZE" -- \
>
>
> I don't understand why, but this is how I fixed the script.
> Should the documentation be fixed?

Yes, thank you! I can confirm that the "rw" kernel cmdline option is indeed
essential (I don't know why) so thanks for finding this before I make a release
with wrong instructions.

Can you share the final version of your script? Maybe I should include it
somewhere in the mmdebstrap packaging.

Thanks!

cheers, josch

[Johannes Schauer Marin Rodrigues]

Quoting Francesco Poli (2022-03-01 23:53:27)
> I performed the first test: I seem to be able to use the resulting image as a
> testbed for autopkgtest!

awesome! Feel free to close this bug then.

> > Can you share the final version of your script? Maybe I should include it
> > somewhere in the mmdebstrap packaging.
>

> Well, I would be happy to see it included in the package.
> A very preliminary version (basically, the first one that produced a
> usable image, after all the failed attempts since the beginning of this
> bug log!) is attached.
>
> As you can see from the various FIXME and TODO comments, there's a lot
> of room for improvement.
> Of course, if you feel like suggesting modifications (especially those
> that implement any of the imagined developments described in the FIXME
> or TODO comments), please go ahead!
>
> Looking forward to receiving your feedback. Bye and thanks for your kind
> assistance!

I used your FIXME and TODO comments to put the instructions from the man page
into a finished script:

https://gitlab.mister-muffin.de/josch/mmdebstrap/src/branch/main/examples/mmdebstrap-autopkgtest-qemu

Mainly there are now no temporary files anymore and everything works in a
single shell pipeline.

I'm afraid that this will not work on architectures other than amd64 and i386
for the reasons given in the comments at the top of the script.

I've put this into the examples directory for now. I'm not sure whether I want
to make this into a script that can be put into PATH and which would then also
need options to set the image size and distribution.

Thanks!

cheers, josch

[Francesco Poli]

On Fri, 04 Mar 2022 12:34:41 +0100 Johannes Schauer Marin Rodrigues wrote:

> Quoting Francesco Poli (2022-03-01 23:53:27)
> > I performed the first test: I seem to be able to use the resulting image as a
> > testbed for autopkgtest!
>
> awesome! Feel free to close this bug then.

Well, I am afraid I'll have to ask for your help one more time.
See below.

>
> > > Can you share the final version of your script? Maybe I should include it
> > > somewhere in the mmdebstrap packaging.

[...]

> > Looking forward to receiving your feedback. Bye and thanks for your kind
> > assistance!
>
> I used your FIXME and TODO comments to put the instructions from the man page
> into a finished script:
>
> https://gitlab.mister-muffin.de/josch/mmdebstrap/src/branch/main/examples/mmdebstrap-autopkgtest-qemu
>
> Mainly there are now no temporary files anymore and everything works in a
> single shell pipeline.

That should be an improvement, thanks.
I was gradually going in the same direction, as well.

>
> I'm afraid that this will not work on architectures other than amd64 and i386
> for the reasons given in the comments at the top of the script.

I see. That's really unfortunate.
I hope it can be fixed in the future.

>
> I've put this into the examples directory for now. I'm not sure whether I want
> to make this into a script that can be put into PATH and which would then also
> need options to set the image size and distribution.

I will comment on your version of the script later on.

For the time being, I tried to test it.
I cannot understand why, but it fails for me:

I: automatically chosen mode: unshare
I: chroot architecture amd64 is equal to the host's architecture
I: automatically chosen format: tar

I: using /dev/shm/mmdebstrap.oJ0dezfu6T as tempdir
[...]
I: installing essential packages...
done
Selecting previously unselected package base-files.
dpkg: regarding ...//base-files_12.2_amd64.deb containing base-files, pre-dependency problem:
base-files pre-depends on awk
awk is not installed.

dpkg: warning: ignoring pre-dependency problem!
(Reading database ... 0 files and directories currently installed.)
Preparing to unpack ...//base-files_12.2_amd64.deb ...
[...]
Processing triggers for libc-bin (2.33-7) ...
dpkg (subprocess): unable to execute installed libc-bin package post-installation script (/var/lib/dpkg/info/libc-bin.postinst): No such file or directory
dpkg: error processing package libc-bin (--install):
installed libc-bin package post-installation script subprocess returned error exit status 2

Errors were encountered while processing:

dash
libgssapi-krb5-2:amd64
login
libpam-modules:amd64
libpam-runtime
libc-bin
E: run_chroot failed: E: env --unset=APT_CONFIG --unset=TMPDIR /usr/sbin/chroot /dev/shm/mmdebstrap.oJ0dezfu6T dpkg --install --force-depends --status-fd=<$fd> /var/cache/apt/archives//base-files_12.2_amd64.deb [...] /var/cache/apt/archives//zlib1g_1%3a1.2.11.dfsg-2_amd64.deb failed

W: listening on child socket failed:

I: removing tempdir /dev/shm/mmdebstrap.oJ0dezfu6T...

E: mmdebstrap failed to run

The most surprising thing is that I basically get the same errors with
my own version of the script.
The same script that had worked yesterday!
Well, actually, what I reported above is taken from the output of the my own version of the script... :-/

Is there any major uninstallability going on in Debian unstable?

My (host) system has had an upgrade that changed the running kernel and
a few other packages:

[INSTALL, DEPENDENCIES] linux-image-5.16.0-3-amd64:amd64 5.16.11-1
[UPGRADE] libbpf0:amd64 1:0.5.0-1 -> 1:0.7.0-2
[UPGRADE] libhivex0:amd64 1.3.21-1+b5 -> 1.3.21-1.1
[UPGRADE] libssh2-1:amd64 1.10.0-2 -> 1.10.0-3
[UPGRADE] libwin-hivex-perl:amd64 1.3.21-1+b5 -> 1.3.21-1.1
[UPGRADE] linux-image-amd64:amd64 5.16.7-2 -> 5.16.11-1
[UPGRADE] linux-libc-dev:amd64 5.16.7-2 -> 5.16.11-1

Do you think that any of these package upgrade could have caused the
regression I am seeing?

Otherwise, what could be the root cause of the issue?

[Johannes Schauer Marin Rodrigues]

Quoting Francesco Poli (2022-03-04 19:22:11)

The problem is dash. I've already fixed it here:

https://salsa.debian.org/debian/dash/-/merge_requests/12

So the problem should fix itself once your mirror has the new dash version.

Thanks!

cheers, josch

[Francesco Poli]

On Sun, 06 Mar 2022 10:24:10 +0100 Johannes Schauer Marin Rodrigues wrote:

> Quoting Francesco Poli (2022-03-05 23:53:47)
> > On Fri, 04 Mar 2022 20:54:51 +0100 Johannes Schauer Marin Rodrigues wrote:
> >
> > [...]

> > > The problem is dash. I've already fixed it here:
> > >
> > > https://salsa.debian.org/debian/dash/-/merge_requests/12
> > >
> > > So the problem should fix itself once your mirror has the new dash version.
> >

> > Yes, I can confirm that the script is working properly today! I am
> > consequently closing this bug report (at last!).
>
> Thank you for sticking around that long.

You are welcome, I am glad that it finally worked out.

> Many bug reporters vanish after their
> initial report but despite this having been quite the ride you kept around
> until the end. That's the kind of bugreporter you want to have as a developer.
> :)

Being a developer myself, I can only agree.

Moreover, as a user, I can say that the assistance you provided has
been quite useful: this is really appreciated. Thanks a lot!

>
> > Thanks for all the help that you have provided since I originally reported
> > this wishlist bug!
> >
> > Comments on your version of the script:
[...]
> > Please consider following these suggestions.
>
> The following? You mean the previous, no? :)

No, I meant something like "Please follow my advice" / "Please follow
my suggestions".

Anyway, all of this is pointless now, since you enhanced the
script! ;-)

>
> The reason why the script has no command line arguments and lives in the
> examples directory is because it is only useful on amd64 and i386. But the
> mmdebstrap package is Architecture:all and will thus be installed on all
> architectures. It would be wrong to install a script into $PATH on an
> architecture where it is useless.
>
> But I now figured out how to use grub instead of extlinux which was limiting us
> to amd64 and i386. Additionally I figured out how to setup grub to do an EFI or
> ieee1275 boot. EFI is required for ARM platforms and ieee1275 for ppc64el.
>
> Here is the finished script:
>
> https://gitlab.mister-muffin.de/josch/mmdebstrap/src/branch/main/mmdebstrap-autopkgtest-build-qemu

I have tested it: it works and I like it (although it generates a
somewhat larger .qcow2 file: 1.3 GB vs. 885 MB).

I strongly encourage you to ship this script in the mmdebstrap package
and to have it installed in one directory within the user's $PATH .

Thanks again! :-)
Bye.

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Francesco Poli (2022-01-13 23:52:02)

> On Mon, 10 Jan 2022 23:18:36 +0100 Johannes Schauer Marin Rodrigues wrote:
> > You could set TMPDIR to a location that has enough space.
>
> I had tried, if you recall.
> But you told me that the temporary directory must be world-writable
> (and preferably with the sticky bit set) and all the directories in the
> path must be world-readable.
>
> This rules out anything within my home directory (setting my home
> directory as world-readable is out of the question).

I just remembered this message of yours and it occurred to me that maybe the
following bit might be interesting to you. If you have a directory like this:

/home/username/tmp

And if /home/username is set to 750, then an unshared process (or any process
other than one from your user or group) will not be able to read
/home/username/tmp either, even if you chmod /home/username/tmp to 1777.

But to fix this, there is another way than making /home/username
world-readable. You can also chmod /home/username to 751. That way, the
executable bit is set on your home directory but the readable bit is not. This
means that other processes are able to access directories below /home/username
if they know their path but they can *not* get a directory listing of
/home/username.

Maybe you already know this in which case, sorry for the noise. :)

In any case I think that this detail about the executable bit versus the
readable bit on directories is not widely known and I should probably add some
documentation to the mmdebstrap man page about this to inform users that the
path to their TMPDIR does *not* need to be world-readable, just
world-executable.

Thanks!

cheers, josch

[Francesco Poli]

On Fri, 11 Mar 2022 09:30:32 +0100 Johannes Schauer Marin Rodrigues
wrote:

[...]

> Quoting Francesco Poli (2022-01-13 23:52:02)

[...]

> > This rules out anything within my home directory (setting my home
> > directory as world-readable is out of the question).

[...]

> You can also chmod /home/username to 751. That way, the
> executable bit is set on your home directory but the readable bit is not. This
> means that other processes are able to access directories below /home/username
> if they know their path but they can *not* get a directory listing of
> /home/username.

Yes, I am aware of the distinction between read and execute permissions
for directories, although it seldom is relevant, since, in the most
common case, directories have either both permissions set or both
unset...

However, it's true that those two permission bits have different effects
on directories.

As for my home directory, I am not in the mood to set it
world-executable.

>
> Maybe you already know this in which case, sorry for the noise. :)

No need to apologize: on the contrary, I would like to thank you for
mentioning this possibility!

>
> In any case I think that this detail about the executable bit versus the
> readable bit on directories is not widely known and I should probably add some
> documentation to the mmdebstrap man page about this to inform users that the
> path to their TMPDIR does *not* need to be world-readable, just
> world-executable.

I agree that this subtle detail should be mentioned in mmdebstrap
documentation. Some users may be unwilling to let a directory become
world-readable, but, in some cases, could accept to let it become
world-executable... So it's good, if the documentation describes this
aspect in the most accurate manner.

Bye!

[Francesco Poli]

On Sun, 6 Mar 2022 19:26:54 +0100 Francesco Poli wrote:

[...]

> I strongly encourage you to ship this script in the mmdebstrap package
> and to have it installed in one directory within the user's $PATH .

Hi again!

Do I understand correctly that mmdebstrap-autopkgtest-build-qemu is
currently [included] in the Debian source package, but not shipped in
the Debian binary package?

$ dpkg -L mmdebstrap | grep qemu

gives no output.

[included]: <https://salsa.debian.org/debian/mmdebstrap/-/blob/master/mmdebstrap-autopkgtest-build-qemu>

Are you going to ship the script in the next version of the binary
package?

Please let me know.
Thanks!

[Johannes Schauer Marin Rodrigues]

Hi Francesco,

Quoting Francesco Poli (2022-03-23 00:09:21)

> Do I understand correctly that mmdebstrap-autopkgtest-build-qemu is currently
> [included] in the Debian source package, but not shipped in the Debian binary
> package?
>
> $ dpkg -L mmdebstrap | grep qemu
>
> gives no output.
>
> [included]: <https://salsa.debian.org/debian/mmdebstrap/-/blob/master/mmdebstrap-autopkgtest-build-qemu>
>
> Are you going to ship the script in the next version of the binary package?

short answer: no.

A bit longer: The perfect is again the enemy of the good.

Full: the problem with the current version of mmdebstrap-autopkgtest-build-qemu
is, that it can only build qemu images for the native architecture. This is
because it relies on guestfish. Guestfish will never be able to operate on
foreign architecture qemu guests. This is because:

- guestfish sets architecture specific options at compile time. This means
that every guestfish binary can only be used for qemu guests of the same
architecture as that binary and this cannot be changed at runtime

- guestfish relies on another program called supermin. Essentially, supermin
assembles a minimal chroot which is then loaded as qemu boots and then
carries out the guestfish operations. Since supermin just copies binaries
from the host, it cannot create foreign chroots either.

This means we have to replace guestfish by something else. I'm not aware that
this already exists so I wrote a proof-of-concepts that does what we need. It
works by first building a kernel and initramfs and then booting qemu with both.
The initramfs contains scripts that partition the disk, copy the rootfs and
install the bootloader:

main script: http://paste.debian.net/1235312/
initramfs-hook: http://paste.debian.net/1235313/
initramfs-script: http://paste.debian.net/1235314/

This works for both, foreign and native architectures. But since running
mmdebstrap to build the initramfs is far slower than supermin, I'd like to
fallback to using guestfish in the native case. So I have to combine above
script with mmdebstrap-autopkgtest-build-qemu.

I'm yet unsure whether I want to make these more general so that they can be
used for other purposes or whether the script should specifically build
autopkgtest qemu images.

A disadvantage is, that this only works for architectures for which qemu knows
how to boot them without kernel and initrd from the outside. This means that
mips* and s390x are not supported and neither are most of the Debian ports
architectures. I don't know how to solve this other than by teaching
autopkgtest-virt-qemu that now it needs three input files: the kernel, the
initrd and the rootfs.

Another disadvantage is, that the output can never be bit-by-bit reproducible
because grub-install is unreproducible.

I've worked on this in context with replacing vmdb2 in autopkgtest-build-qemu
so that sbuild-qemu-create (maintained by Christian Kastner) can be run without
superuser privileges. We've tried to approach the vmdb2 author but Lars is
reluctant to include such drastic changes:
https://gitlab.com/larswirzenius/vmdb2/-/issues/62

So I'll probably release yet another https://wiki.debian.org/SystemBuildTools
because the existing solutions don't do what I want. The closest is probably
debos which can be run without being root because everything is done inside
qemu. But it also doesn't solve the hard problem of installing a bootloader,
leaving it to the user: https://github.com/go-debos/debos/issues/137

I also still need a name for this tool.

Thanks!

cheers, josch

[Christian Kastner]

Johannes,

thank you for this enlightening update!

You're far ahead of any ideas I've been having (some of which I
attribute to Franscesco), but I'd nevertheless like to contribute a few
thoughts just in case.

On 2022-03-23 12:43, Johannes Schauer Marin Rodrigues wrote:
> Quoting Francesco Poli (2022-03-23 00:09:21)

> Full: the problem with the current version of mmdebstrap-autopkgtest-build-qemu
> is, that it can only build qemu images for the native architecture. This is
> because it relies on guestfish. Guestfish will never be able to operate on
> foreign architecture qemu guests. This is because:
>
> - guestfish sets architecture specific options at compile time. This means
> that every guestfish binary can only be used for qemu guests of the same
> architecture as that binary and this cannot be changed at runtime
>
> - guestfish relies on another program called supermin. Essentially, supermin
> assembles a minimal chroot which is then loaded as qemu boots and then
> carries out the guestfish operations. Since supermin just copies binaries
> from the host, it cannot create foreign chroots either.

This is unfortunate, as otherwise guestfish would be universal, but it's
understandable.

I initially wondered whether foreign architecture guestfish/supermin
binaries couldn't be run through qemu-user-static, but that solution
would be just far too hacky I think.

> This means we have to replace guestfish by something else. I'm not aware that
> this already exists so I wrote a proof-of-concepts that does what we need. It
> works by first building a kernel and initramfs and then booting qemu with both.
> The initramfs contains scripts that partition the disk, copy the rootfs and
> install the bootloader:
>
> main script: http://paste.debian.net/1235312/
> initramfs-hook: http://paste.debian.net/1235313/
> initramfs-script: http://paste.debian.net/1235314/

This is really neat!

> This works for both, foreign and native architectures. But since running
> mmdebstrap to build the initramfs is far slower than supermin, I'd like to
> fallback to using guestfish in the native case. So I have to combine above
> script with mmdebstrap-autopkgtest-build-qemu.
>
> I'm yet unsure whether I want to make these more general so that they can be
> used for other purposes or whether the script should specifically build
> autopkgtest qemu images.
>
> A disadvantage is, that this only works for architectures for which qemu knows
> how to boot them without kernel and initrd from the outside. This means that
> mips* and s390x are not supported and neither are most of the Debian ports
> architectures. I don't know how to solve this other than by teaching
> autopkgtest-virt-qemu that now it needs three input files: the kernel, the
> initrd and the rootfs.

Since these are QEMU options, I think autopkgtest-virt-qemu's existing
--qemu-options could be used to supply kernel/initrd/rootfs?

> Another disadvantage is, that the output can never be bit-by-bit reproducible
> because grub-install is unreproducible.
>
> I've worked on this in context with replacing vmdb2 in autopkgtest-build-qemu
> so that sbuild-qemu-create (maintained by Christian Kastner) can be run without
> superuser privileges. We've tried to approach the vmdb2 author but Lars is
> reluctant to include such drastic changes:
> https://gitlab.com/larswirzenius/vmdb2/-/issues/62
>
> So I'll probably release yet another https://wiki.debian.org/SystemBuildTools
> because the existing solutions don't do what I want. The closest is probably
> debos which can be run without being root because everything is done inside
> qemu. But it also doesn't solve the hard problem of installing a bootloader,
> leaving it to the user: https://github.com/go-debos/debos/issues/137

I think this is a fantastic idea, not just because sbuild-qemu would be
a major beneficiary but because there is a general gap here: there
currently is no tool that can easily create images (1) without root and
(2) for arbitrary architectures.

Having such a tool would enable many upstreams to easily test their
software on other architectures, e.g. via GitHub Actions. PowerPC and
RISC-V are interesting contenders but only the fewest projects test
them. However, our buildds and debci discover issues all the time.

Additionally, it's really useful to have an on-demand, throwaway
porterbox when you need one. I probably use sbuild-qemu-boot more than
sbuild-qemu, and the former is just a QEMU launcher. I use it even for
native architectures because it's basically a container-like workspace
where I can do Bad Stuff without fear of polluting or trashing my system.

I've run out of ideas how to easily solve this, other than what
Francesco suggested (e.g. create a base with FAI or preseed.cfg, then
finalize by running commands in the VM just as sbuild-qemu-update does
it). Really looking forward to what you are working on.

If you need a test user, I'll happily volunteer!

Best,
Christian

[Johannes Schauer Marin Rodrigues]

Quoting Christian Kastner (2022-03-23 21:36:59)

> On 2022-03-23 12:43, Johannes Schauer Marin Rodrigues wrote:
> > Quoting Francesco Poli (2022-03-23 00:09:21)
> > Full: the problem with the current version of mmdebstrap-autopkgtest-build-qemu
> > is, that it can only build qemu images for the native architecture. This is
> > because it relies on guestfish. Guestfish will never be able to operate on
> > foreign architecture qemu guests. This is because:
> >
> > - guestfish sets architecture specific options at compile time. This means
> > that every guestfish binary can only be used for qemu guests of the same
> > architecture as that binary and this cannot be changed at runtime
> >
> > - guestfish relies on another program called supermin. Essentially, supermin
> > assembles a minimal chroot which is then loaded as qemu boots and then
> > carries out the guestfish operations. Since supermin just copies binaries
> > from the host, it cannot create foreign chroots either.
>
> This is unfortunate, as otherwise guestfish would be universal, but it's
> understandable.
>
> I initially wondered whether foreign architecture guestfish/supermin
> binaries couldn't be run through qemu-user-static, but that solution
> would be just far too hacky I think.

I think how "hacky" it is depends on how you look at it. The advantage is, that
then you only have to maintain the "native" codepath which you run under
emulation if you want a foreign arch binary. I just tried it out and it seems
to work as expected. Big downside: running qemu (from guestfish) inside qemu
(from qemu-user-static) is *reeeeaaaaalllyyyyyy sloooooooooow*. XD

But I think it's a good idea to have for the sake of symmetry. We'd then have:

- guestfish
- fastest method when creating a native image
- slowest method when creating a foreign image
- initramfs
- medium speed for both native and foreign images

Then the new tool could have a --method switch with possible values:

- "auto" choose guestfish if installed for native images and initramfs
otherwise
- "guestfish" forces guestfish for native and foreign
- "initramfs" forces initramfs for native and foreign

> > A disadvantage is, that this only works for architectures for which qemu knows
> > how to boot them without kernel and initrd from the outside. This means that
> > mips* and s390x are not supported and neither are most of the Debian ports
> > architectures. I don't know how to solve this other than by teaching
> > autopkgtest-virt-qemu that now it needs three input files: the kernel, the
> > initrd and the rootfs.
>
> Since these are QEMU options, I think autopkgtest-virt-qemu's existing
> --qemu-options could be used to supply kernel/initrd/rootfs?

Yes, that might work. Though from reading the autopkgtest-virt-qemu man page I
think this will not support paths with spaces. On the other hand, the man page
also says that additional arguments can be supplied via a file that contains
one argument per line.

I'm not so sure whether it's a fantastic idea. First I want to evaluate other
existing tools like debos and see if they can do the job or not. I want to
resist the NIH syndrome.

Another question that I'm still pondering is how much configuration this new
tool should allow. I'm tempted to make it just fit for the use-case of
autopkgtest backend or porterbox or sbuild-qemu backend just to keep it simple.

And lastly, the name is also still an open question.

Alas, I'll probably not get to working on this much in the next two weeks.

Lets see. :)

Thanks!

cheers, josch

[Christian Kastner]

On 2022-04-02 07:58, Johannes Schauer Marin Rodrigues wrote:
> Quoting Christian Kastner (2022-03-23 21:36:59)

>> I initially wondered whether foreign architecture guestfish/supermin
>> binaries couldn't be run through qemu-user-static, but that solution
>> would be just far too hacky I think.
>
> I think how "hacky" it is depends on how you look at it. The advantage is, that
> then you only have to maintain the "native" codepath which you run under
> emulation if you want a foreign arch binary. I just tried it out and it seems
> to work as expected.

Well, that's cool to hear!

> Big downside: running qemu (from guestfish) inside qemu (from
> qemu-user-static) is *reeeeaaaaalllyyyyyy sloooooooooow*. XD

How slow? On my end, IIRC sbuild-qemu-create takes 2mins to build a
native image, and 12-13mins to build a foreign one.

However, that's wall-time, for a process that can run unattended. Having
a one-time, slow, non-interactive process might not be that bad,
compared to needing root.

> But I think it's a good idea to have for the sake of symmetry. We'd then have:
>
> - guestfish
> - fastest method when creating a native image
> - slowest method when creating a foreign image
> - initramfs
> - medium speed for both native and foreign images
>
> Then the new tool could have a --method switch with possible values:
>
> - "auto" choose guestfish if installed for native images and initramfs
> otherwise
> - "guestfish" forces guestfish for native and foreign
> - "initramfs" forces initramfs for native and foreign

Sounds good.

>> Since these are QEMU options, I think autopkgtest-virt-qemu's existing
>> --qemu-options could be used to supply kernel/initrd/rootfs?
>
> Yes, that might work. Though from reading the autopkgtest-virt-qemu man page I
> think this will not support paths with spaces. On the other hand, the man page
> also says that additional arguments can be supplied via a file that contains
> one argument per line.

Indeed, I just checked and autopkgtest-virt-qemu just calls .split() on
the string. The file workaround should be fine, though.

> I'm not so sure whether it's a fantastic idea. First I want to evaluate other
> existing tools like debos and see if they can do the job or not. I want to
> resist the NIH syndrome.

Full ACK. Hence the focus on extending vmdb2 if possible, or using
guestfish or debos, but I myself don't favor any particular tool. My
first choice would simply be the tool that accomplishes it.

> Another question that I'm still pondering is how much configuration this new
> tool should allow. I'm tempted to make it just fit for the use-case of
> autopkgtest backend or porterbox or sbuild-qemu backend just to keep it simple.

Indeed, I have asked myself the same question a number of times.

It's tempting to make this as flexible as possible, but in the end, it's
main use cases are build and test environments. From experience, this
means that one frequently needs to configure hardware (e.g. number of
CPUs), or modify the build environment (configure external repos,
pre-install packages, copy files e.g. SSH keys).

So it can be helpful to facilitate a few common uses, but it's probably
overkill to allow for all, and it's probably also not needed: after all,
one can always boot the image, and modify interactively.

> And lastly, the name is also still an open question.

The hardest part :D

> Alas, I'll probably not get to working on this much in the next two weeks.

Best of luck! I wish I could do more to help, but still swamped at the
moment, with little time for Debian.

Best,
Christian