Bug#1056556: mmdebstrap: mmdebstrap error resolving installed packages

[Dima Kogan]

Package: mmdebstrap
Version: 1.4.0-1
Severity: normal

Hi. I'm seeing a failure that I understand very well, but yet don't know
how to debug or fix. Any suggestions would be appreciated.

I'm making an Ubuntu/focal image that has a bunch of stuff installed,
and can serve as a base for development. This runs on arm64. There're a
number of ugly external APT repos that have semi-broken packages, but it
should all still work.

I define the stuff I want to install into the image with a meta-package.
tst-libopencv.equivs:

Source: tst-libopencv
Section: misc
Priority: optional
Standards-Version: 3.9.2

Package: tst-libopencv
Version: 1
Maintainer: Dima Kogan <dko...@debian.org>

Depends: ros-noetic-cv-bridge,
libopencv-dev (<< 4.5)

Architecture: arm64
Description: Test

And I build the meta-package:

equivs-build -aarm64 tst-libopencv.equivs

And I can use mmdebstrap to create a base image with this package
installed:

mmdebstrap \
--verbose \
--architectures=arm64 \
--hook-dir=/usr/share/mmdebstrap/hooks/file-mirror-automount \
--include ./tst-libopencv_1_arm64.deb \
focal \
tst.tar.gz \
"deb [trusted=yes] http://ports.ubuntu.com/ubuntu-ports/ focal main restricted universe multiverse" \
"deb [trusted=yes] http://ports.ubuntu.com/ubuntu-ports/ focal-updates main restricted universe multiverse" \
"deb [trusted=yes] http://ports.ubuntu.com/ubuntu-ports/ focal-backports main restricted universe multiverse" \
"deb [trusted=yes] http://ports.ubuntu.com/ubuntu-ports/ focal-security main restricted universe multiverse" \
"deb [trusted=yes] http://packages.ros.org/ros/ubuntu focal main"

This works great. ros-noetic-cv-bridge is an external-apt-repo package.
It Depends: libopencv-dev. Ubuntu/focal ships 4.2, so the requirement
libopencv-dev (<< 4.5) in the meta-package is satisfied.

OK. But let's say I want to add another, also-heinous external repo into
the mix, and I do this:

mmdebstrap \
--verbose \
--architectures=arm64 \
--hook-dir=/usr/share/mmdebstrap/hooks/file-mirror-automount \
--include ./tst-libopencv_1_arm64.deb \
focal \
tst.tar.gz \
"deb [trusted=yes] http://ports.ubuntu.com/ubuntu-ports/ focal main restricted universe multiverse" \
"deb [trusted=yes] http://ports.ubuntu.com/ubuntu-ports/ focal-updates main restricted universe multiverse" \
"deb [trusted=yes] http://ports.ubuntu.com/ubuntu-ports/ focal-backports main restricted universe multiverse" \
"deb [trusted=yes] http://ports.ubuntu.com/ubuntu-ports/ focal-security main restricted universe multiverse" \
"deb [trusted=yes] https://repo.download.nvidia.com/jetson/common r35.4 main" \
"deb [trusted=yes] https://repo.download.nvidia.com/jetson/t234 r35.4 main" \
"deb [trusted=yes] http://packages.ros.org/ros/ubuntu focal main"

This is the same command, but I also make some nvidia packages
available. THAT repo ships its own copy of libopencv-dev: version
4.5.xxxxx. When building the image I explicitly do NOT want it to pick
up that version, but to use the normal Ubuntu/focal ones: that
restriction in the meta-package should do that for me. There's no reason
this shouldn't work, and I can easily create this situation with some
apt commands after I chroot into the image. But mmdebstrap cannot create
this image: the above command fails:

The following packages have unmet dependencies:
tst-libopencv : Depends: libopencv-dev (< 4.5) but 4.5.4-8-g3e4c170df4 is to be installed
E: Unable to correct problems, you have held broken packages.
E: setup failed: E: apt-get -o Dir::Bin::dpkg=env -o DPkg::Options::=--unset=TMPDIR -o DPkg::Options::=dpkg -o DPkg::Chroot-Directory=/tmp/mmdebstrap.rIZdCWvgpA --yes install -oAPT::Status-Fd=<$fd> -oDpkg::Use-Pty=false /tmp/tst-libopencv_1_arm64.deb ?narrow(?or(?archive(^focal$),?codename(^focal$)),?architecture(arm64),?and(?or(?priority(required),?priority(important)),?not(?essential))) failed

Suggestions?

Thanks.



-- System Information:
Debian Release: trixie/sid
APT prefers unstable
APT policy: (800, 'unstable'), (500, 'unstable-debug'), (500, 'stable')
Architecture: amd64 (x86_64)
Foreign Architectures: armhf, armel

Kernel: Linux 6.4.0-3-amd64 (SMP w/4 CPU threads; PREEMPT)
Locale: LANG=C, LC_CTYPE=C.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages mmdebstrap depends on:
ii apt 2.7.3
ii perl 5.36.0-8
ii python3 3.11.4-5+b1

Versions of packages mmdebstrap recommends:
pn arch-test <none>
pn fakechroot <none>
ii fakeroot 1.32.1-1
ii gpg 2.2.40-1.1
ii libdistro-info-perl 1.5
ii libdpkg-perl 1.21.22
ii mount 2.39.2-1
ii uidmap 1:4.13+dfsg1-1+b1

Versions of packages mmdebstrap suggests:
pn apt-transport-tor <none>
ii apt-utils 2.7.3
ii binfmt-support 2.2.2-2
ii ca-certificates 20230311
ii debootstrap 1.0.131
ii distro-info-data 0.58
ii dpkg-dev 1.21.22
pn genext2fs <none>
ii perl-doc 5.36.0-8
pn qemu-user <none>
ii qemu-user-static 1:8.0.4+dfsg-3+b1
pn squashfs-tools-ng <none>
ii systemd 254.1-3

-- no debconf information

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Dima Kogan (2023-11-23 02:54:16)

> Hi. I'm seeing a failure that I understand very well, but yet don't know
> how to debug or fix. Any suggestions would be appreciated.
>
> I'm making an Ubuntu/focal image that has a bunch of stuff installed,
> and can serve as a base for development. This runs on arm64. There're a
> number of ugly external APT repos that have semi-broken packages, but it
> should all still work.

damn, my "moon robot stuff needs fixing" alarm clock just woke me up at 3:39 AM
in the morning for yet another mmdebstrap bug. Lets go! ;)

> This is the same command, but I also make some nvidia packages
> available. THAT repo ships its own copy of libopencv-dev: version
> 4.5.xxxxx. When building the image I explicitly do NOT want it to pick
> up that version, but to use the normal Ubuntu/focal ones: that
> restriction in the meta-package should do that for me. There's no reason
> this shouldn't work, and I can easily create this situation with some
> apt commands after I chroot into the image. But mmdebstrap cannot create
> this image: the above command fails:
>
> The following packages have unmet dependencies:
> tst-libopencv : Depends: libopencv-dev (< 4.5) but 4.5.4-8-g3e4c170df4 is to be installed
> E: Unable to correct problems, you have held broken packages.
> E: setup failed: E: apt-get -o Dir::Bin::dpkg=env -o DPkg::Options::=--unset=TMPDIR -o DPkg::Options::=dpkg -o DPkg::Chroot-Directory=/tmp/mmdebstrap.rIZdCWvgpA --yes install -oAPT::Status-Fd=<$fd> -oDpkg::Use-Pty=false /tmp/tst-libopencv_1_arm64.deb ?narrow(?or(?archive(^focal$),?codename(^focal$)),?architecture(arm64),?and(?or(?priority(required),?priority(important)),?not(?essential))) failed
>
> Suggestions?

If you want to install everything with just a single command (i.e. just use
--include with your meta-package instead of using multiple --customize-hooks
that run apt install with the right options) then you need to use apt pinning.
The reason for that is that apt will only consider candidate versions for
installation. It picks candidate versions by choosing the package with the
highest pinning priority and in case of a tie, chooses the highest version. So
even if you have one package with a Depends: A (< 2) and you have repositories
configured with both A (= 1) and A (= 2), apt will only look at A (= 2) and
then fail to install the package due to the A (< 2) version restriction. Apt
will not look at A (= 1) to satisfy the dependency. This is just a limitation
in apt.

What I do in another project (incidentally also building arm64 images) is to
add this to my mmdebstrap invocation:

--setup-hook='{ echo "Package: XXX"; echo "Pin: origin \"YYY\""; echo "Pin-Priority: 1"; } > "$1"/etc/apt/preferences.d/mypinnings.pref'


In your case, XXX should probably be libopencv-dev and YYY should be
repo.download.nvidia.com. With that being set, libopencv-dev coming from
repo.download.nvidia.com will be put to a very low pin priority (1) and thus
not get selected by apt.

Hope this helps!

cheers, josch

[Dima Kogan]

Hi. I tried to do that apt pinning today, as you suggested. It still
fails in the same way as before:

$ mmdebstrap ....
....
I: installing remaining packages inside the chroot...
....
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:

ros-noetic-cv-bridge : Depends: libopencv-dev but it is not installable
ros-noetic-grid-map-filters : Depends: libopencv-dev but it is not installable
ros-noetic-image-geometry : Depends: libopencv-dev but it is not installable

E: Unable to correct problems, you have held broken packages.

The preferences file is there, but it isn't obviously doing anything. I
did some debugging just now to try to figure out why, and I'm reminded
of

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1036929

Ideally the workflow I would like is

1. mmdebstrap does stuff. When it runs a command, it prints out EXACTLY
what it does, in a way that I can copy/paste it, and get the same
output

2. If it fails to do something, it drops me into a shell. Where I can
paste the command to reproduce the problem, and then poke around to
fix it

This probably isn't 100% possible here; but how would you debug this
otherwise?

I'm attaching a patch that does some of (1.) above. This uses
String::ShellQuote to quote all the arguments so that the command string
can be pasted. I'm guessing you'd want to do some of that differently,
so it isn't super thorough.

OK. Then I gave myself a shell in a spot that (I think?) sits right
before the failing "apt-get install":

mmdebstrap \
--essential-hook 'echo $$1; bash -i' \
....

And pasting the command didn't work as I had hoped:

root@fatty:# apt-get -o Dir::Bin::dpkg=env -o DPkg::Options::=--unset=TMPDIR -o DPkg::Options::=dpkg -o DPkg::Chroot-Directory=/tmp/mmdebstrap.i1wpW0WLMS --yes install -oDpkg::Use-Pty=false tst-libopoencv '?narrow(?or(?archive(^focal$),?codename(^focal$)),?architecture(arm64),?and(?or(?priority(required),?priority(important)),?not(?essential)))'

E: Could not open lock file /var/lib/dpkg/lock-frontend - open (13: Permission denied)
E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), are you root?

Any idea what the difference is? Here I'm running the external
"apt-get". Is the "apt-get install" supposed to happen inside the
chroot? There is no "apt-get" binary there yet.

If I can quickly and repeatedly reproduce the failure, I can "apt-cache
policy libopencv-dev", and I can see if it's trying to use the
preferences file and such. Maybe I mistyped something.

If the above diagnostic sequence cannot work, what would?

Thanks much

[Johannes Schauer Marin Rodrigues]

Hi Dima,

qick top post as I'm about to head out for work and I don't have access to my
Debian mail there. I'll write you a more complete reply in ~12 hours when I'm
back.

Until then, the short answer is: you cannot re-run that apt-get in a shell
because of the file descriptors that are passed, the pseudo terminal setup,
proc handling, /dev/ptmx setup etc. This is also why it's not so easy (or even
possible?) to copypaste the command and run it in a shell. A lot of things that
mmdebstrap does are not possible (I think) in a shell because a careful dance
of foreground and background processes talking with each other over pipes is
constructed in many cases.

It is possible to run the command you attempted to run from a shell but the
command will be a bit different and thus it will not do exactly the same thing.

What I would try in your situation is to not list the packages you want to
install via --include but instead run mmdebstrap like this:

mmdebstrap ... --variant=apt --chrooted-customize-hook=bash unstable /dev/null

This will give you a shell inside a minimal chroot and from there you can mess
around with apt pinning and run arbitrary apt-get install commands. Once you
are happy, try to copy what you did in the shell interactively into --include
options and --hooks.

Hope that helps!

cheers, josch

Quoting Dima Kogan (2023-11-28 00:02:43)

[Dima Kogan]

Hi josch.

I sorta expected that there was extra complexity here that made
debugging difficult. It's unfortunate.

> mmdebstrap ... --variant=apt --chrooted-customize-hook=bash unstable /dev/null

Would that work, though?

--chrooted-customize-hook isn't in the manpage

--customize-hook runs after everything was installed (so past where
the failure was happening here)

--essential-hook was running at the right time, but the "apt-get"
executable wasn't available

In any case, I figured out my specific problem by creating a similar
scenario on a native arm64 box. I was naming the pinning file xxxx.conf
instead of xxxx.pref which apparently matters. On the arm64 box this
produced a clear error message ("apt" told me to rename the file). But
with mmdebstrap there was no specific error at all, as you saw. Any idea
why?

Thanks for all the help.

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Dima Kogan (2023-11-28 17:25:01)

> I sorta expected that there was extra complexity here that made
> debugging difficult. It's unfortunate.

to be fair, I have yet to see a problem involving apt pinning that was not
difficult. If the wrong package that got installed did not make you immediately
think of apt pinning, then this would've also happened without mmdebstrap on
any regular system.

> > mmdebstrap ... --variant=apt --chrooted-customize-hook=bash unstable /dev/null
>
> Would that work, though?

Yes. Did you try it and it did not work? What was the error message?

> --chrooted-customize-hook isn't in the manpage

Correct. That's on purpose.

> --customize-hook runs after everything was installed (so past where
> the failure was happening here)

That's why I suggested *not* to put your packages into --include but running
"apt-get install" manually once you have bash inside the chroot.

> --essential-hook was running at the right time, but the "apt-get"
> executable wasn't available

Yes, because apt is not essential. That's why I recommended using
--variant=apt.

> In any case, I figured out my specific problem by creating a similar
> scenario on a native arm64 box. I was naming the pinning file xxxx.conf
> instead of xxxx.pref which apparently matters.

Yes, in the man page of apt_preferences it says: "The files have either no or
"pref" as filename extension". In one of my last mails to you I also suggested
you add:

--setup-hook='{ echo "Package: XXX"; echo "Pin: origin \"YYY\""; echo "Pin-Priority: 1"; } > "$1"/etc/apt/preferences.d/mypinnings.pref'

Notice that I used the correct filename extension.

> On the arm64 box this produced a clear error message ("apt" told me to
> rename the file). But with mmdebstrap there was no specific error at all, as
> you saw. Any idea why?

Which apt command produced the error? I also don't think it was an error. It
was only a warning, right? Did you get it for "apt-get update" or for "apt-get
install"?

Thanks!

cheers, josch

[Dima Kogan]

Johannes Schauer Marin Rodrigues <jo...@debian.org> writes:

>> > mmdebstrap ... --variant=apt --chrooted-customize-hook=bash unstable /dev/null
>>
>> Would that work, though?
> Yes. Did you try it and it did not work? What was the error message?

No :) I wanted to read about what it did first. I tried it just now,
though. With the --included metapackage it has the same behavior as
before: complains that it can't install libopencv-dev.

If I don't ask it to --install the problematic package, intending to
manually poke apt, then I can't do that: the problematic package is
nowhere to be found. It was originally on disk locally, but without
--include, it was never copied into the bind mount.

I do want to have a metapackage: this allows the metapackage to be
updated in the future, and have users be able to "apt update && apt
upgrade".

I guess I could do this differently for testing. By making the
metapackage available in an apt server, and using an undocumented
option. That's a heavy lift though. If I was so expert to know to do
these things, I probably wouldn't need to debug stuff in the first place

>> In any case, I figured out my specific problem by creating a similar
>> scenario on a native arm64 box. I was naming the pinning file xxxx.conf
>> instead of xxxx.pref which apparently matters.
>
> Yes, in the man page of apt_preferences it says: "The files have either no or
> "pref" as filename extension". In one of my last mails to you I also suggested
> you add:
>
> --setup-hook='{ echo "Package: XXX"; echo "Pin: origin \"YYY\""; echo
> "Pin-Priority: 1"; } > "$1"/etc/apt/preferences.d/mypinnings.pref'
>
> Notice that I used the correct filename extension.

Yep. I liked my extension better, but didn't realize that the name was
significant.

>> On the arm64 box this produced a clear error message ("apt" told me to
>> rename the file). But with mmdebstrap there was no specific error at all, as
>> you saw. Any idea why?
>
> Which apt command produced the error? I also don't think it was an error. It
> was only a warning, right? Did you get it for "apt-get update" or for "apt-get
> install"?

Great questions. I just tried it again:

$ sudo apt update

....
N: Ignoring file 'mypinnings.conf' in directory '/etc/apt/preferences.d/' as it has an invalid filename extension

So I don't know what the answer is. But this felt undebuggable, and I
wish I could figure this stuff without sinking many hours into it or
asking you every time.

Thanks for the help, as always.

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Dima Kogan (2023-12-01 05:11:19)

> Johannes Schauer Marin Rodrigues <jo...@debian.org> writes:
> >> Would that work, though?
> > Yes. Did you try it and it did not work? What was the error message?
>
> No :) I wanted to read about what it did first. I tried it just now,
> though. With the --included metapackage it has the same behavior as
> before: complains that it can't install libopencv-dev.

yes, which is why I said not to use --include when trying this.

> If I don't ask it to --install the problematic package, intending to
> manually poke apt, then I can't do that: the problematic package is
> nowhere to be found. It was originally on disk locally, but without
> --include, it was never copied into the bind mount.

Sure, you just have to do that yourself.

> I do want to have a metapackage: this allows the metapackage to be updated in
> the future, and have users be able to "apt update && apt upgrade".

Yes, I think you should keep the metapackage.

> I guess I could do this differently for testing. By making the metapackage
> available in an apt server, and using an undocumented option. That's a heavy
> lift though. If I was so expert to know to do these things, I probably
> wouldn't need to debug stuff in the first place

You don't need to use an undocumented option. Running
--chrooted-customize-hook=bash is nearly the same as running
--customize-hook='chroot "$1" bash' but the latter is much harder to type due
to quoting and special chars so I suggested to do the former. But nothing stops
you from sticking to the documented methods.

> > Yes, in the man page of apt_preferences it says: "The files have either no
> > or "pref" as filename extension". In one of my last mails to you I also
> > suggested you add:
> >
> > --setup-hook='{ echo "Package: XXX"; echo "Pin: origin \"YYY\""; echo
> > "Pin-Priority: 1"; } > "$1"/etc/apt/preferences.d/mypinnings.pref'
> >
> > Notice that I used the correct filename extension.
>
> Yep. I liked my extension better, but didn't realize that the name was
> significant.

Maybe only try out different things *after* you have it working? ;)

> > Which apt command produced the error? I also don't think it was an error.
> > It was only a warning, right? Did you get it for "apt-get update" or for
> > "apt-get install"?
>
> Great questions. I just tried it again:
>
> $ sudo apt update
>
> ....
> N: Ignoring file 'mypinnings.conf' in directory '/etc/apt/preferences.d/' as it has an invalid filename extension
>
> So I don't know what the answer is.

By default, mmdebstrap does not print the output of the commands it runs. It
does that though when something goes wrong. So if "apt install" fails, then you
get its output. In your case, you missed a "note" (not even a warning) in the
"apt update" output. You would've seen that if you had run mmdebstrap with the
--verbose option.

> But this felt undebuggable, and I wish I could figure this stuff without
> sinking many hours into it or asking you every time.

I still wonder what we can learn from all of this.

You initially suggested mmdebstrap to drop down to a shell and print a command
for the user to re-run. Lets assume that this were possible (I don't think it
is feasible). In that case, you still would not've known how to proceed or that
it is the apt pinning that is at fault. If you had known that it was apt
pinning (as we know now), then you could've just added this to your mmdebstrap
invocation to debug this:

--essential-hook='chroot "$1" apt-cache policy libopencv-dev'

But even if you had and you had seen that the pin priority was wrong, how could
you have proceeded? At that point (and before installing
tst-libopencv_1_arm64.deb via --include) you could've given yourself an
interactive shell by adding (and lets stick to the documented options this
time):

--essential-hook='chroot "$1" bash'

You could've then maybe figured out what was going on as you can then make
changes and run "apt-cache policy" until it looks good. But maybe you wouldn't
have know it because you still would not've seen the apt "note" from above
about the filename extension because even though you encountered an error, you
did not run mmdebstrap with --verbose and I think it is common practice to
increase the verbosity of a program first when something goes wrong.

So given all of this, I don't think your initial suggestion of adding a
facility to drop to a shell and re-run a command in shell would've fixed your
problem. It would not've given you the additional knowledge that apt pinning is
at fault, you still would have to do the bind-mount yourself manually (or would
you expect that bind mounting to be translated into a shell command as well?)
and you still would not've seen that apt "note" about your preferences file
name.

So I'm at a loss at what to take away from this situation for mmdebstrap.

I have no problem helping you with this and it doesn't bother me but I also
don't see anything that is to be learned from all of this.

Thanks!

cheers, josch

[Dima Kogan]

Hi

Johannes Schauer Marin Rodrigues <jo...@debian.org> writes:

> By default, mmdebstrap does not print the output of the commands it runs. It
> does that though when something goes wrong. So if "apt install" fails, then you
> get its output. In your case, you missed a "note" (not even a warning) in the
> "apt update" output. You would've seen that if you had run mmdebstrap with the
> --verbose option.

No. I was running mmdebstrap --verbose, and the note wasn't in the
output. I only saw the note when I set up a similar situation in a real
arm64 install (no schroot, no mmdebstrap), and tried to "apt update"
there. mmdebstrap passing on the note would have helped.

> You initially suggested mmdebstrap to drop down to a shell and print a command
> for the user to re-run. Lets assume that this were possible (I don't think it
> is feasible). In that case, you still would not've known how to proceed or that
> it is the apt pinning that is at fault.

> <snip>

If a shell was available, you can do lots of quick experiments quickly,
and narrow down the problem quickly. I routinely use tools like sysdig
to report all system-wide syscalls, and that output is enough to figure
out lots of problems. You can use it without the shell too, without
reproducing the problem in isolation, but it's harder to interpret the
logs.

> So given all of this, I don't think your initial suggestion of adding a
> facility to drop to a shell and re-run a command in shell would've fixed your
> problem.

Maybe. Maybe not. Can we agree that the capability to do this in sbuild
is extremely useful, at least? I find it extremely valuable.

> I have no problem helping you with this and it doesn't bother me but I
> also don't see anything that is to be learned from all of this.

OK. Hopefully I don't need to bug you many more times.

Thanks a lot for the help!