Bug#1033802: dropbear-initramfs: sleep and cat not found

[William Desportes]

Package: dropbear-initramfs
Severity: important

I am trying to sort out this bug, but the machine is blocked in an endless loop

It does /scripts/init-bottom

And then the monitor displays in an endless loop:
/scripts/init-premount/dropbear: line 339: sleep: not found
/scripts/init-premount/dropbear: line 149: cat: not found

I think the source code is at: https://salsa.debian.org/debian/dropbear/-/blob/debian/2022.83-1/debian/initramfs/scripts/init-bottom/dropbear#L48

So it's maybe waiting for a non starting dropbear.


I used https://wiki.debian.org/DropBear as an example and did set: DROPBEAR_OPTIONS="-p 2222 -s"

[Guilhem Moulin]

Control: tag -1 unreproducible moreinfo

On Sat, 01 Apr 2023 at 18:36:47 +0200, William Desportes wrote:
> I am trying to sort out this bug, but the machine is blocked in an endless loop
>
> It does /scripts/init-bottom
>
> And then the monitor displays in an endless loop:
> /scripts/init-premount/dropbear: line 339: sleep: not found
> /scripts/init-premount/dropbear: line 149: cat: not found

Please file bugs using the reportbug(1) template and provide a debug
output, there is nothing actionable here. The affected version isn't
even specified let alone loose dependencies…

FWIW sleep and cat are provided by busybox which dropbear-initramfs
Depends on. Also we have an autopkgtest for dropbear-initramfs (and
encrypted root FS), and the issue is not reproducible there.

If there is nothing blocking between the init-premount and init-bottom
scripts this might be the same as https://bugs.debian.org/1015810 .

> I used https://wiki.debian.org/DropBear as an example and did set: DROPBEAR_OPTIONS="-p 2222 -s"

FYI this page is not maintained by the src:dropbear maintainer. I'll
even delete it since it contains misleading and even potentially
dangerous information.

Authoritative information can be found in
/usr/share/doc/dropbear-initramfs/README.initramfs and the config file.

--
Guilhem.

[William Desportes]

Thanks for the quick reply, that's very appreciated

>Please file bugs using the reportbug(1) template and provide a debug
>output, there is nothing actionable here. The affected version isn't
>even specified let alone loose dependencies…

Well, it's complicated to use reportbug on machine that does not boot at time of reporting this.
It's pure Debian 12.
Will report more about the dependencies.
With the DI alpha2 I managed to recovery boot into a shell and remove the package dropbear-initramfs
It of course makes me able to boot afterwards, but when I install it back the error is back.

>FWIW sleep and cat are provided by busybox which dropbear-initramfs
>Depends on. Also we have an autopkgtest for dropbear-initramfs (and
>encrypted root FS), and the issue is not reproducible there.

I agree but in reality they seem not to exist, I am not sure how it's even possible.

>If there is nothing blocking between the init-premount and init-bottom
>scripts this might be the same as https://bugs.debian.org/1015810 .

I will have a look to this bug

>FYI this page is not maintained by the src:dropbear maintainer. I'll
>even delete it since it contains misleading and even potentially
>dangerous information.

Can you be more specific?, I updated some of it yesterday. The updating keys seems to be useful.

I will most updates as soon as I manage to pin point this. It's very very weird, my first thought is that dropbear maybe does not start

[Guilhem Moulin]

On Sun, 02 Apr 2023 at 10:54:59 +0200, William Desportes wrote:
> Can you be more specific?, I updated some of it yesterday. The
> updating keys seems to be useful.

See the NEWS entry for 2015.68-1, /etc/ssh and the initramfs image have
different access control so blindly suggesting to convert key materiel
from one to the other is a bad idea. The page also had misleading/
confusing information between dropbear and dropbear-initramfs.

I'll be happy to try to improve
/usr/share/doc/dropbear-initramfs/README.initramfs if needed, or even
create an HTML page under .pages.debian.net if online material is
desired, but I think the wiki shouldn't be used as a brain dump that way
without coordination with the maintainer (duplication of existing
documentation shipped in the source package tend to rot and diverge over
time).

--
Guilhem.

[William Desportes]

My Debug did some small progress the other day, and can confirm I walked into https://bugs.debian.org/1015810

> /scripts/init-premount/dropbear: line 300: can't open '/run/net-*.conf': No such file or directory

That said, It also says
> g: eth0: SI0CGIFINDEX: No such device
> g: no devices to configure

So there is maybe more to my bug.

> I'll be happy to try to improve /usr/share/doc/dropbear-initramfs/README.initramfs if needed, or even
create an HTML page under .pages.debian.net if online material is
desired

I would be glad to help ! :)

> but I think the wiki shouldn't be used as a brain dump that way
without coordination with the maintainer (duplication of existing
documentation shipped in the source package tend to rot and diverge over
time).

I agree, I was quite deprecated and past out.

Can you please revert your change to the wiki, it's not user friendly and will help nobody since it does not link to a proper page.
Many internet sources do still link to the wiki page.
The page could be emptied and have a link to a .pages.debian.net

But the wiki page should redirect to somewhere. Maybe the README.initramfs on Debian sources or Salsa ?

> See the NEWS entry for 2015.68-1, /etc/ssh and the initramfs image have
different access control so blindly suggesting to convert key materiel
from one to the other is a bad idea.  The page also had misleading/
confusing information between dropbear and dropbear-initramfs.

I did not apply the key part as I saw it generated keys by itself
and dropbear-initramfs worked perfectly without key instructions on my RaspberryPi 4B

--
William Desportes

[Guilhem Moulin]

On Wed, 05 Apr 2023 at 23:11:36 +0200, William Desportes wrote:
> My Debug did some small progress the other day, and can confirm I walked into https://bugs.debian.org/1015810
>
>> /scripts/init-premount/dropbear: line 300: can't open '/run/net-*.conf': No such file or directory
>
> That said, It also says
>> g: eth0: SI0CGIFINDEX: No such device
>> g: no devices to configure
>
> So there is maybe more to my bug.

Doesn't look like a dropbear-initramfs bug, possibly a missing module if
the device isn't exposed at initramfs stage.

--
Guilhem.

[William Desportes]

> Doesn't look like a dropbear-initramfs bug, possibly a missing module if
the device isn't exposed at initramfs stage.

I re-installed the machine, at this point I no longer have the sleep and cat missing bug.
Not sure If I want to try having it back, with cryptsetup it does not like rescue mode initramfs updates.

That said I have a new one (I did not install the basic system utilities):

It now says: init-premount/dropbear: line 366: ipconfig: not found

And right after the message about net-*.conf

The system does not have ipconfig installed, I would say this is a bug that should be fixed in the dropbear script.
Or added a as a package dependency/requirement ?

--
William Desportes

[Guilhem Moulin]

On Thu, 06 Apr 2023 at 18:56:49 +0200, William Desportes wrote:
> with cryptsetup it does not like rescue mode initramfs updates.

Hm? Installing cryptsetup-initramfs, and letting it unlock devices
(incl. those holding the root FS) at early boot stage, definitely
doesn't prevent rescue mode or getting an initramfs shell.

> That said I have a new one (I did not install the basic system utilities):
>
> It now says: init-premount/dropbear: line 366: ipconfig: not found
>
> And right after the message about net-*.conf

dropbear-initramfs doesn't call ipconfig or configure the network
directly. All it does is: call initramfs-tools' configure_networking()
at init-premount stage, and bring the interface down at init-bottom
stage.

> The system does not have ipconfig installed,

What do you mean? Your main system (outside) initramfs stage might lack
/usr/bin/ipconfig but that is irrelevant. What matters is that the
binary is available at initramfs stage.

> I would say this is a bug that should be fixed in the dropbear script.
> Or added a as a package dependency/requirement ?

Nope. The /usr/bin/ipconfig binary from the initramfs image comes from
/usr/lib/klibc/bin/ipconfig which dropbear-initramfs indirectly has a
hard Depends: on…

Anyway, the ‘/run/net-*.conf’ glob and the ipconfig call both come
initramfs-tools' configure_networking(). As I wrote before if these
fail it's most likely because there is nothing blocking at initramfs
stage, so execution is handed over to init(1) before
configure_networking() has a chance to terminate (it runs in the
background). This, again, smells like #1015810.

--
Guilhem.

[Guilhem Moulin]

On Thu, 06 Apr 2023 at 23:15:59 +0200, Guilhem Moulin wrote:
> On Thu, 06 Apr 2023 at 18:56:49 +0200, William Desportes wrote:
>> The system does not have ipconfig installed,
>
> What do you mean? Your main system (outside) initramfs stage might lack

Misplaced parenthesis, that should have been “Your main system (outside
initramfs stage) might lack …”

> Anyway, the ‘/run/net-*.conf’ glob and the ipconfig call both come
> initramfs-tools' configure_networking(). As I wrote before if these
> fail it's most likely because there is nothing blocking at initramfs
> stage, so execution is handed over to init(1) before
> configure_networking() has a chance to terminate (it runs in the
> background). This, again, smells like #1015810.

What are you trying to do by the way? Are you installing
dropbear-initramfs on that machine not because you *need* to get a
remote initramfs shell in order to boot, but to have a way to remotely
access access the machine should something break at early boot stage?

--
Guilhem.

[William Desportes]

> Hm? Installing cryptsetup-initramfs, and letting it unlock devices
(incl. those holding the root FS) at early boot stage, definitely
doesn't prevent rescue mode or getting an initramfs shell.

I mean that the Debian installer after some rounds of getter a rescue
chroot to debug this bug
did start not finding "sda_crypt" and then the boot was falling into a
shell and the system not booting.
At this point I re-installed the full system. Because clearly there was
an open and mounted /dev/mapper/sda_crypt

Anyway, it's off topic I think ^^

> What are you trying to do by the way? Are you installing
dropbear-initramfs on that machine not because you *need* to get a
remote initramfs shell in order to boot, but to have a way to remotely
access access the machine should something break at early boot stage?

On this machine, I was planning to do the same config that works
perfectly on my RaspberryPi to remote SSH unlock the disk.
But on this machine (it has two ethernet ports) It does not work and I
have only bugs.
For now I have a screen so I can unlock the disk with a keyboard.

> Nope. The /usr/bin/ipconfig binary from the initramfs image comes from
/usr/lib/klibc/bin/ipconfig which dropbear-initramfs indirectly has a
hard Depends: on…

Well maybe, but clearly this is not the hole truth I bet. Right after
boot&unlock and (user login?) it prints the missing ipconfig missing
message.
So the script is also called out of initramfs.

> Anyway, the ‘/run/net-*.conf’ glob and the ipconfig call both come
initramfs-tools' configure_networking(). As I wrote before if these
fail it's most likely because there is nothing blocking at initramfs
stage, so execution is handed over to init(1) before
configure_networking() has a chance to terminate (it runs in the
background). This, again, smells like #1015810.

I have to dig on this aspect. Different that for the RaspberryPi setup,
this machine has nothing hooked to it's ethernet ports.


For the record, this is my RaspberryPi setup:

/etc/dropbear/initramfs/dropbear.conf

DROPBEAR_OPTIONS="-I 180 -j -k -p 2222 -s"

/etc/initramfs-tools/initramfs.conf

IP=192.168.4.45::192.168.4.1:255.255.255.0:rpi:eth0:off

[Guilhem Moulin]

On Thu, 13 Apr 2023 at 23:16:15 +0200, William Desportes wrote:
> Right after boot&unlock and (user login?) it prints the missing
> ipconfig missing message.

Just to confirm, you unlock (at initramfs stage) using keyboard + screen
right, not remotely using dropbear SSH right? Because at that point the
initramfs hasn't got network access right?

> So the script is also called out of initramfs.

Again if is nothing blocking at initramfs stage then execution is handed

over to init(1) before configure_networking() has a chance to terminate
(it runs in the background).

This is going in circles, I have no idea how to explain better than I
did in https://bugs.debian.org/1015810#10 :

| However if you don't have BOOT=nfs then initramfs-tools-core's
| configure_networking() runs asynchronously starting from init-premount
| stage and might not give up before the execution is handed over to the
| normal system. dropbear-initramfs' init-bottom scripts only wait for
| 60s by default but configure_networking() might take much longer (180s
| timeout for the device, up to 10s waiting time for udev to settle, then
| exponential backoff of 2+3+4+6+9+16+25+36+64+100=265s for the network
| configuration).
|
| Since 2020.81-2 the init-bottom timeout is configurable with
| DROPBEAR_SHUTDOWN_TIMEOUT and you can set it to a value that exceeds the
| typical configure_networking() duration on your system to be sure that
| there is leftover process passed init-bottom stage.

Can you please try to set DROPBEAR_SHUTDOWN_TIMEOUT=300 or so, as I
pointed in my first replied, before insisting there is a bug or missing
dependencies?

--
Guilhem.

[Guilhem Moulin]

On Wed, 29 Nov 2023 at 14:11:09 +0100, William Desportes wrote:
> I had put an interface name: ens9.123 thinking it would take the VLAN tag.
> But it triggered the crash. Removing the ".123" fixes it.

That's #1015287.

As written in msg#42 dropbear-initramfs doesn't configure the network by
itself; all it does is calling initramfs-tools' configure_networking().
And that function currently doesn't support VLAN tags, which is what
the aforementioned wishlist bug is about.

> That said, sleep and cat are still not into the initramfs image.

What makes you say that? What's the output of `lsinitramfs /initrd.img | grep -e/{sleep,cat}`?

You still haven't replied whether setting DROPBEAR_SHUTDOWN_TIMEOUT=300
fixes this or not, and I still have reasons to believe it's the
(non-)issue described earlier in this bug.

--
Guilhem.

[William Desportes]

> As written in msg#42 dropbear-initramfs doesn't configure the network
> by
> itself; all it does is calling initramfs-tools' configure_networking().
> And that function currently doesn't support VLAN tags, which is what
> the aforementioned wishlist bug is about.

ack. I will try to use an other interface name that does not exist.

> What makes you say that? What's the output of `lsinitramfs /initrd.img
> | grep -e/{sleep,cat}`?

usr/lib/modules/6.1.0-13-amd64/kernel/drivers/net/usb/catc.ko
usr/bin/sleep
usr/bin/cat

> You still haven't replied whether setting DROPBEAR_SHUTDOWN_TIMEOUT=300
> fixes this or not, and I still have reasons to believe it's the
> (non-)issue described earlier in this bug.

ack, will try this.


I will try to boot a VM and do very minimal steps to get to the final
bug.