Bug#1000239: Rescue system won't find root partition, but insists on /usr

[TomK]

Package: debian-installer
Version: 20210731+deb11u1_amd64

Errors, "No suitable shell found on /dev/sda1"

Cycling through every partition results in what should be /usr being selected as the root partition. This is useless for a rescue syatem, because there some commands missing.

I tried to mount the root partition on a mount point I made inside the installer, but there was a problem with permissions. I can't perfectly remember. I tried 'sudo', but it isn't available in the installer.

I was attempting to reinstall grub, to get the system to boot. I have also used the rescue syatem to change a lost password. It appears this bug may apply only to a gpt partition table and a fat32 boot partition (efi?).

I've had this same problem with 2 systems, a lappy and a desktop. I can always work around it. I don't understand what could be going on, for lack of experience. I've only been a Debian user since Woody was in testing.

It's easy to reproduce. Do an expert install with defaults, but partition with gpt. Boot the system with the install media, launch a rescue shell, and try to open a root shell. An incorrect device should be identified as '/'. Perhaps this occurs only when /usr is on a different partition that '/'.

But it is bothersome, because it essentially removes 90% of the functionality from the rescue system. The install media used to be my goto for rescue. But now I've found this very annoying problem. If you require more info, I can boot the dvd again and record exact errors.

I'm on my tablet, so I don't have access to reportbug. Thanks for the help! You guys are the best!

[Steve McIntyre]

Hi Tom,

Just to be sure, can you expand on this "expert install with defaults"
please? Exactly what partition layout are you using? Reading the rest
of your mail, I get the impresion that you're putting /usr on a
different fs from the rootfs?

As well as working on the rescue setup, I've personally used it quite
a bit over the years and so long as you identify the rootfs ok it
normally works just fine in my experience. Hence I'm curious to see
how your setup may be causing a different experience...

--
Steve McIntyre, Cambridge, UK. st...@einval.com
"War does not determine who is right - only who is left."
-- Bertrand Russell

[Nicholas D Steeves]

Control: tag -1 + moreinfo

Hi Steve,

I've written my reply assuming this isn't a usermerged system, because
if it is one, then I wonder if this is a usrmerge-related bug. Ie: I
wonder if a splitting / and /usr onto different partitions is never
supported on usrmerged systems. Alternatively, maybe the rescue mode
isn't usrmerge-ready?

Hi Tom,

TomK <fore...@milwpc.com> writes:

> Package: debian-installer
> Version: 20210731+deb11u1_amd64
>
> Errors, "No suitable shell found on /dev/sda1"
>

[snip]

> I was attempting to reinstall grub, to get the system to boot. I have also used the rescue syatem to change a lost password. It appears this bug may apply only to a gpt partition table and a fat32 boot partition (efi?).
>
> I've had this same problem with 2 systems, a lappy and a desktop. I can always work around it. I don't understand what could be going on, for lack of experience. I've only been a Debian user since Woody was in testing.
>
> It's easy to reproduce. Do an expert install with defaults, but partition with gpt. Boot the system with the install media, launch a rescue shell, and try to open a root shell. An incorrect device should be identified as '/'. Perhaps this occurs only when /usr is on a different partition that '/'.
>

Given the information you provided, this is what I suspect may be
occurring:

1. You have a EFI system
2. On an EFI system, /dev/sda1 is usually the ESP (EFI System Partition)
3. The ESP is not the rootfs, and it won't contain /bin/init, /bin/sh, etc.
4. Therefore choosing /dev/sda1 as the rootfs will never boot
5. due to 'Errors, "No suitable shell found on /dev/sda1"'

If you're dual booting with another OS, the rootfs might be on sda2.
Would you please share note the error[s] when selecting sda2, sda3, or
whichever partition ?

Other relevant info is if LUKS (encrypted partition) or an configuration
using LVM was selected.

Optional: Feel free to unset the "moreinfo" tag when providing this
information (see the first line of this email for the model, and change
the "+" to a "-" in your reply).

P.S. It's likely that I won't be able to follow-up on this bug until
after the holidays,

Regards,
Nicholas

[Nicholas D Steeves]

Control: severity -1 serious
Control: tags = confirmed

CCing the release team, and CTTE because I don't know who else is
tracking issues related to the usrmerge effort. I've consciously chosen
not to pour gasoline on the flame war by CCing anyone else (nor will I
contact anyone else about the existence of this bug).

Steps I used to try to reproduce:

1. Downloaded debian-testing-amd64-netinst.iso 2021-12-03 16:21 408M
2. Installed to EFI-enabled qemu eg:
kvm -bios /usr/share/ovmf/bios.bin -m 2G \
-hda debian-separate-usr-sda.raw \
-cdrom debian-testing-amd64-netinst.iso
3. Guided partitioning with separate /home, then changed the mount point
to /usr. Partition layout is:
sda1: ESP fat32
sda2: / ext4
sda3: swap swap
sda4: /usr ext4
4. Selected only "Standard System Utilities"
5. Rebooted

Result: SUCCESS

Then, to test the rescue functionality:

1. I discovered qemu+OVMF boot order is broken without changing the
command to: kvm -L /usr/share/ovmf/ -m 2G -boot menu=on \
-hda /scratch/debian-separate-usr-sda.raw \
-cdrom /scratch/debian-testing-amd64-netinst.iso
2. Selected sda2
3. Yes, mount /boot/efi
4. Execute a shell in /dev/sda2
5. No usable shell was found on your root file system (/dev/sda2)
6. Changed virtual terminal
7. cd /target && ls bin
ls: bin: No such file or directory

Result: FAILED
Conclusion: This is a usrmerged system, and the rescue system does not
support usermerged systems.

The options are, as I see them, ranked from least to most work-hours:

1. Debian isn't yet ready for usrmerge. Revert to normal system installation.
2. Reassign to src:rescue, and fix the rescue system.
a) before chrooting, test for the presence of /target/bin/sh
b) if /bin/sh is not found, either emit error to the user and present a
dialogue for selecting /usr partition, or
c) parse /target/etc/fstab, and attempt to mount other partitions
d) b & c will be difficult to implement when attempting to accommodate
the heterogeneity of possible MD, LUKS, and LVM layouts, not to mention
the complications introduced by the possibility of a user-configured
btrfs subvolume name "@usr" on any valid device. Fstab parsing might
make the btrfs case easier with:
i) Display a dialogue selector if a btrfs partition is detected
The dialogue will list all detected subvolumes
ii) If the user cannot find a subvolume for "@usr", then
iii) Allow the user to escape to the partition selection screen, and
iv) Unmount the partition
3. Disallow configuring of a mount for "/usr", and *Prominently* declare that
Debian no longer supports separating /usr from /, declare this in *many
places*, and reply to bugs on this topic for many years. I put this one
last because I believe the cost to work-hours is unbounded, and
because I believe there may be a negative social cost associated with
this action. Also, if Fedora/RHEL/SUSE/Ubuntu support a separate /usr
partition, then this action could make Debian look inferior.


Regards,
Nicholas

[Wouter Verhelst]

FWIW, Debian was the last holdout in still supporting a separate /usr
partition amongst major distributions, so that bit is irrelevant...

--
w@uter.{be,co.za}
wouter@{grep.be,fosdem.org,debian.org}

[Michael Biebl]

Am 03.12.2021 um 22:08 schrieb Nicholas D Steeves:
> 2. Reassign to src:rescue, and fix the rescue system.

Looks like a duplicate of
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=769738

[Simon McVittie]

(Speaking only on my own behalf, not on behalf of the TC, here)

On Fri, 03 Dec 2021 at 16:08:24 -0500, Nicholas D Steeves wrote:
> 1. Debian isn't yet ready for usrmerge

Merged /usr is not actually the problem here, although it exacerbates what
appears to be a pre-existing bug in the rescue mode[0]. The root cause is
that since Debian 9 [1][2], the "/usr-like" parts of the root filesystem
are no longer guaranteed to be self-contained: important shared libraries[3]
and executables have been gradually moving into /usr for a while, and
stretch was the point at which this was declared to be officially allowed.

Merged /usr makes this very noticeable because it's the most extreme
version of that process, where *literally everything* "/usr-like" (most
notably /bin/sh) moves into /usr. When /usr is a separate filesystem, that
leaves a root filesystem that potentially contains only /etc, symlinks and
mount points, which is certainly not enough to be useful to chroot into.

> 2. Reassign to src:rescue, and fix the rescue system.

I think this will have to be the answer. See also [0].

> 3. Disallow configuring of a mount for "/usr"

That would be unfortunate, since one of the things enabled by merged
/usr is the ability to keep all "/usr-like" content on a separate /usr
filesystem that is mounted read-only during normal operation, remounting
it rw only for system updates[4], while leaving /etc and /var rw (and
potentially offering the ability to reformat the partition with /etc
and /var, and then repopulate it via systemd-tmpfiles or similar, as a
"factory reset" mechanism).

smcv

[0] https://bugs.debian.org/769738 (opened in 2014)
[1] https://www.debian.org/releases/stretch/amd64/release-notes/ch-information.en.html#late-mounting-usr
[2] and possibly earlier: that release note was documenting current practice
rather than describing a new change
[3] for example about half the libraries in `ldd /sbin/cryptsetup`
[4] which potentially happen atomically across a reboot, as seen in ostree,
or while rebooted into a special boot mode, as with
systemd.offline-updates(7)

[Kenneth Parker]

I would like to add myself to the Testing part of this process. 

Some may have noticed some "static" from me Yesterday, when I, somehow messed up the "Install Desktop from Network" options, on two qemu Installs of Bookworm. 

I assume that, Bookworm would be the earliest place where most of the Changes would be made to the Rescue System? (Though I have one or two Bullseye Systems that are Expendable also). 

 

[0] https://bugs.debian.org/769738 (opened in 2014)
[1] https://www.debian.org/releases/stretch/amd64/release-notes/ch-information.en.html#late-mounting-usr
[2] and possibly earlier: that release note was documenting current practice
    rather than describing a new change
[3] for example about half the libraries in `ldd /sbin/cryptsetup`
[4] which potentially happen atomically across a reboot, as seen in ostree,
    or while rebooted into a special boot mode, as with
    systemd.offline-updates(7)

Speaking of SystemD, are there, yet any good "SystemD for Dummies" books out there?  That is something I would pay Money for. 

Thanks! 

Kenneth Parker 

[Philip Hands]

Simon McVittie <sm...@debian.org> writes:

>> 2. Reassign to src:rescue, and fix the rescue system.
>
> I think this will have to be the answer. See also [0].

I'm certainly not against fixing this issue, but I thought I'd point out
that even as things are, its pretty trivial to get into rescue mode.

Having just tested it with a separate /usr, all one needs to do is
select the actual root partition as you'd expect, then when prompted to
execute a shell, select the second option (Execute a shell in the
installer environment), then the trick is to mount the /usr partition by
hand onto /target/usr e.g.:

# mount /var/vfa4 /target/mnt

Having done that one can exit that shell, and then select the first
option (Execute shell in /dev/...) and you're away.

Cheers, Phil.
--
|)| Philip Hands [+44 (0)20 8530 9560] HANDS.COM Ltd.
|-| http://www.hands.com/ http://ftp.uk.debian.org/
|(| Hugo-Klemm-Strasse 34, 21075 Hamburg, GERMANY

[Pascal Hambourg]

Le 03/12/2021 à 22:08, Nicholas D Steeves a écrit :
>
>
> c) parse /target/etc/fstab, and attempt to mount other partitions

The rescue system already offers to do it for separate /boot and
/boot/efi, so why couldn't it do for separate /usr too ?

[Steve McIntyre]

That's exactly what I'm about to do.

--
Steve McIntyre, Cambridge, UK. st...@einval.com

"Managing a volunteer open source project is a lot like herding
kittens, except the kittens randomly appear and disappear because they
have day jobs." -- Matt Mackall

[Steve McIntyre]

On Sat, Dec 04, 2021 at 10:42:29PM +0000, Steve McIntyre wrote:
>On Sat, Dec 04, 2021 at 11:37:28PM +0100, Pascal Hambourg wrote:
>>Le 03/12/2021 à 22:08, Nicholas D Steeves a écrit :
>>>
>>>
>>> c) parse /target/etc/fstab, and attempt to mount other partitions
>>
>>The rescue system already offers to do it for separate /boot and /boot/efi,
>>so why couldn't it do for separate /usr too ?
>
>That's exactly what I'm about to do.

In fact, it needed more work than that - the code chrooted into
/target and ran mount there. That didn't work for a separate /usr. But
I've refactored the code and made things work more cleanly inside d-i.

I'm pondering backporting the same fix for future bullseye and buster
point releases.

--
Steve McIntyre, Cambridge, UK. st...@einval.com

Dance like no one's watching. Encrypt like everyone is.
- @torproject

[Cyril Brulebois]

Steve McIntyre <st...@einval.com> (2021-12-06):

> In fact, it needed more work than that - the code chrooted into
> /target and ran mount there. That didn't work for a separate /usr. But
> I've refactored the code and made things work more cleanly inside d-i.
>
> I'm pondering backporting the same fix for future bullseye and buster
> point releases.

I haven't looked at the changes yes (nor do I have a timetable allowing
me to do that soon) but maybe it'd be best to let those changes be
tested, i.e. in some bookworm alpha 1, before considering backporting
them to earlier stable releases? No objections per se otherwise.


Cheers,
--
Cyril Brulebois (ki...@debian.org) <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant

[TomK]

This sounds correct, based on my experience with the bug. I suppose now there are zero ways to definitively determine which partition is actually root. So maybe a hidden flag (empty) file in root might do the trick.

But the fact that /usr can be automounted, but nothing else can be manually mounted afterward is also a problem. If there was a separate script to do the chroot part, and some manual interface by which to mount; perhaps the mount command in the rescue system PATH, the whole thing would be more flexible during the transition to ro mount for /usr in future Debian versions.

Presently it appears there will be several possibilities for the conditions of root and /usr, so no definite means cannot be devised to always do the correct thing. Even if a flag file is used to identify root, that might not be what rescue requires.

In the near future I think the flag file would work, and then /usr could be mounted manually, as once was the case. But perhaps at some point mount would need to be a static binary, to account for libraries moved to /usr.

I'm not well versed on Debian policy, so just ignore my input if it makes no sense. If you find it irritating, it's my ignorance at work, nothing intentional.

[Philip Hands]

TomK <fore...@milwpc.com> writes:

> This sounds correct, based on my experience with the bug.

> I suppose now there are zero ways to definitively determine which
> partition is actually root. So maybe a hidden flag (empty) file in
> root might do the trick.

I'd expect that looking for e.g. /etc/fstab on a file system would
pretty reliably tell you that it's a Linux root filesystem. One could
also/alternatively look for the presence of /sys and /proc as
directories.

> But the fact that /usr can be automounted, but nothing else can be manually mounted afterward is also a problem.

I'm assuming that by "automounted" you mean: automatically mounted by
the rescue system.

If your setup is unusual, then you should probably expect to have to do
some work when rescuing it, which for hard to guess additional
partitions is going to be going into the installer's shell, and mounting
things yourself, which doesn't strike me as particularly difficult (but
should be documented, if it isn't already).

Is it a problem if /home or /usr/share are left unmounted during rescue?
I would say no, unless you've done something unwise, like moving
/usr/lib to /home/oops_I_ran_out_of_room_on_usr/lib, and symlinked it
back to /usr/lib.

> If there was a separate script to do the chroot part, and some manual
> interface by which to mount; perhaps the mount command in the rescue
> system PATH, the whole thing would be more flexible during the
> transition to ro mount for /usr in future Debian versions.

mount is definitely in the rescue system's PATH, since (as I mentioned
before) one could always work round this problem by dropping into the
installer's shell and mounting /usr by hand before then using the
to-be-rescued system's shell.

Actually, I just spun up the latest netinst (with the /usr mount fix) in
rescue mode, and not only does it successfully mount a separate /usr
(well done Steve :-) ), but it has mount in the PATH, and I am able to
umount & mount /boot/efi, so I'm not sure what you're on about here.

> I'm not well versed on Debian policy, so just ignore my input if it
> makes no sense. If you find it irritating, it's my ignorance at work,
> nothing intentional.

No worries, but I think you need to fact-check your assumptions in
future.

[Pascal Hambourg]

Le 08/12/2021 à 10:49, Philip Hands a écrit :
>
> Is it a problem if /home or /usr/share are left unmounted during rescue?

/usr/share contains architecture-independent files for many programs
such as bash, grub, os-prober, debconf, dpkg, initramfs-tools...

How common is it to have a separate /usr/share and what is the rationale
for it ?

[Philip Hands]

Not common at all, I'd guess.

Back in the days when /usr was not needed to boot, I have occasionally
moved part or all of /usr/share (most often just /usr/share/doc) onto
another partition and then put a symlink in, in order to make space when
it turned out that /usr was filling up and it was too painful to resize.

If doing that makes things break during a rescue attempt these days, it
seems fairly likely to break during normal boot, so doing that seems
likely to be the reason for running the rescue system, in which case
you're going to have to mount wherever /usr/share is now yourself, and
you really ought to remember how you just broke the system, so that
seems like no great hardship.

[Steve McIntyre]

On Wed, Dec 08, 2021 at 11:36:47PM +0100, Philip Hands wrote:
>Pascal Hambourg <pas...@plouf.fr.eu.org> writes:
>
>> Le 08/12/2021 à 10:49, Philip Hands a écrit :
>>>
>>> Is it a problem if /home or /usr/share are left unmounted during rescue?
>>
>> /usr/share contains architecture-independent files for many programs
>> such as bash, grub, os-prober, debconf, dpkg, initramfs-tools...
>>
>> How common is it to have a separate /usr/share and what is the rationale
>> for it ?
>
>Not common at all, I'd guess.
>
>Back in the days when /usr was not needed to boot, I have occasionally
>moved part or all of /usr/share (most often just /usr/share/doc) onto
>another partition and then put a symlink in, in order to make space when
>it turned out that /usr was filling up and it was too painful to resize.
>
>If doing that makes things break during a rescue attempt these days, it
>seems fairly likely to break during normal boot, so doing that seems
>likely to be the reason for running the rescue system, in which case
>you're going to have to mount wherever /usr/share is now yourself, and
>you really ought to remember how you just broke the system, so that
>seems like no great hardship.

Nod. A separate /usr filesystem is a configuration that was supported
well by d-i and Debian for a number of years, hence I agreed that it
was worth improving rescue-mode to explicitly support it. I *could*
also be convinced that we should do similar for a separate /var. Other
more exotic configurations are not as important IMHO:

1. People configuring with *many* filesystems that they may want
can run mount manually inside /target once things have started

2. The main point of rescue mode (IMHO) is to get a system up and
running *normally*, particularly if it doesn't boot
currently. Hence the focus on /boot, boot/efi and now /usr. I
believe we should not try to extend that list willy-nilly.

--
Steve McIntyre, Cambridge, UK. st...@einval.com

"...In the UNIX world, people tend to interpret `non-technical user'
as meaning someone who's only ever written one device driver." -- Daniel Pead

[TomK]

In commercial and industrial applications It is common to have discrete partitions for /, /boot, /home, /usr swap and /var. The rationale for doing so lies in three areas. /var can and does fill up with log files frequently. Having it separate has long been a means to prevent / from filling up, and to permit resizing /var without taking down the system.

Many times hardware failure will destroy a partition, and it's much less work to recover if 'everything' does not reside on a single partition. Judicious use of disk space dictatates splitting large disks into smaller partitions, which function more efficiently.

If there is a /usr partition, /usr/share will be part of it. So, separate /usr/share is exactly as common s/as separate /usr. For home use none of this matters, because it is trivial to reinstall the OS. Although, my personal lappy has five partitions on the disk.

/usr/share has architecture independent files which are architecture independent only because they are not binaries. No documentation is architecture dependent! The reader may be, but not the actual documentation, as is found in /usr/share/.

-------- Original Message --------
From: Pascal Hambourg <pas...@plouf.fr.eu.org>
Sent: December 8, 2021 11:41:11 AM UTC
To: Philip Hands <ph...@hands.com>, 100...@bugs.debian.org, TomK <fore...@milwpc.com>
Subject: Re: Bug#1000239: Rescue system won't find root partition, but insists on /usr

[Pascal Hambourg]

Le 09/12/2021 à 00:25, Steve McIntyre a écrit :
>
> Nod. A separate /usr filesystem is a configuration that was supported
> well by d-i and Debian for a number of years, hence I agreed that it
> was worth improving rescue-mode to explicitly support it. I *could*
> also be convinced that we should do similar for a separate /var. Other
> more exotic configurations are not as important IMHO:
>
> 1. People configuring with *many* filesystems that they may want
> can run mount manually inside /target once things have started
>
> 2. The main point of rescue mode (IMHO) is to get a system up and
> running *normally*, particularly if it doesn't boot
> currently. Hence the focus on /boot, boot/efi and now /usr. I
> believe we should not try to extend that list willy-nilly.

I mostly agree. Rescue mode serves two main purposes :
- Execute a shell in $RESCUE_ROOTDEV
- Reinstall the GRUB boot loader

As Philip pointed out, if splitting the hierarchy in some way breaks
executing a shell in rescue mode, it seems fairly likely to break normal
boot too. So IMO, the former does not need to mount more than what the
initramfs does in order to run a usable shell (/, /dev, /proc, /run,
/sys, /usr). Then people may mount whatever they need, for instance by
running "mount -a".

However, the latter has more requirements.
- It executes grub-install which requires a separate /boot and /boot/efi
to be mounted, and maybe /usr/share/grub.
- Sometimes I create a separate /boot/grub filesystem when I want GRUB
to be independent from the rest of the system. So you may consider
mounting it too.
- When the user chooses "Force GRUB installation to the EFI removable
media path", 81grub-efi-force-removable executes debconf-set-selections.
I guess this requires a separate /var to be mounted.

Cosmetic note about rescue-mode.templates : the two messages displayed
before entering a shell contain "If you need any other file systems
(such as a separate "/usr"), you will have to mount those yourself."
Since /usr can now be mounted before executing the shell, shouldn't the
messages be updated to change "/usr" to something else such as "/var" ?

[Pascal Hambourg]

Le 09/12/2021 à 23:29, TomK a écrit :
>
> If there is a /usr partition, /usr/share will be part of it. So,
> separate /usr/share is exactly as common s/as separate /usr.

No, by "separate /usr/share" I (and Philip I guess, please correct me if
I am wrong) mean "separate from /usr".

> /usr/share has architecture independent files which are architecture
> independent only because they are not binaries. No documentation is
> architecture dependent!

/usr/share does not contain only documentation, but also shell scripts
and data files for many programs.

[Nicholas D Steeves]

Thank you for the discussion! I found it educational and motivating,
and hope that everyone found the same. It's really refreshing to see
this. Sorry for the naivety in my analysis and in the delay replying;
I've been drained/busy, but I read each of your emails carefully, wanted
to reply to each of them, and chose not to CC everyone because that
would be noisy.

Steve, thank you for the quick fix :-) Reply follows inline:

Steve McIntyre <st...@einval.com> writes:

> On Sat, Dec 04, 2021 at 10:42:29PM +0000, Steve McIntyre wrote:
>>On Sat, Dec 04, 2021 at 11:37:28PM +0100, Pascal Hambourg wrote:
>>>Le 03/12/2021 à 22:08, Nicholas D Steeves a écrit :
>>>>
>>>>
>>>> c) parse /target/etc/fstab, and attempt to mount other partitions
>>>
>>>The rescue system already offers to do it for separate /boot and /boot/efi,
>>>so why couldn't it do for separate /usr too ?
>>
>>That's exactly what I'm about to do.
>
> In fact, it needed more work than that - the code chrooted into
> /target and ran mount there. That didn't work for a separate /usr. But
> I've refactored the code and made things work more cleanly inside d-i.
>

I had also attempted a fix at the same time, but my approach wasn't
nearly as nice as yours. In particular, I appreciate how your fix
produces insight into debian-installer/rescue function, architecture,
and style, and I hope to say thank you more substantially with an
eventual MR for btrfs subvol support :-) It will be much less ugly and
"tacked on" as a result of studying your changes!

Regards,
Nicholas