Bug#1018730: lvm2: Initramfs does not activate root LVs if VG is incomplete since 2.03.15 or 2.03.16, boot failure

[Melvin Vermeeren]

Package: lvm2
Version: 2.03.16-1
Severity: important
X-Debbugs-Cc: verm...@vermwa.re

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Dear Maintainer,

Since LVM 2.03.15 or possibly 2.03.16 my system fails to boot during initramfs
because the root LVM LV is not activated. I did some digging and confirmed this
is caused by the new method of using udev rules, which have a big flaw.

In /lib/udev/rules.d/69-lvm.rules a VG is only activated once the VG is fully
complete. However in initramfs my HDD arrays are not yet activated because they
need keyfile stored on root partition for automatic activation. This means the
VG is never complete (missing PVs) so it is never activated, meaning the root LV
is never activated at all, resulting in boot failure with initramfs shell.

I have the following storage stack in use on this system:
Disk partition -> integrity -> mdadm -> luks -> lvm -> ext4
Integrity is not part of Debian, I submitted patches a while ago at:
https://salsa.debian.org/cryptsetup-team/cryptsetup/-/merge_requests/22

My LVM setup is the following.
The fc520 contains root, swap LV and all integritysetup meta LVs.
The nas4 and ex12 PVs are exclusively used by hdd LV.

root@verm-r4e:~# pvs
PV VG Fmt Attr PSize PFree
/dev/mapper/md_ex12_0_crypt verm-r4e lvm2 a-- 10.91t 0
/dev/mapper/md_fc520_0_crypt verm-r4e lvm2 a-- 1.81t 256.77g
/dev/mapper/md_nas4_0_crypt verm-r4e lvm2 a-- <3.64t <565.54g
root@verm-r4e:~# vgs
VG #PV #LV #SN Attr VSize VFree
verm-r4e 3 9 0 wz--l- 16.36t 822.31g
root@verm-r4e:~# lvs
LV VG Attr LSize
hdd verm-r4e -wi-ao---- 14.00t
is_boot_0_meta verm-r4e -wi-ao---- 8.00m
is_boot_1_meta verm-r4e -wi-ao---- 8.00m
is_ex12_0_meta verm-r4e -wi-ao---- 11.41g
is_ex12_1_meta verm-r4e -wi-ao---- 11.41g
is_nas4_0_meta verm-r4e -wi-ao---- 4.14g
is_nas4_1_meta verm-r4e -wi-ao---- 4.14g
root verm-r4e -wi-ao---- 1.50t
swap verm-r4e -wi-ao---- 32.00g

Although my integrity setup is not common it's very common for servers with
many encrypted disks is to have something like this:

hdd -> mdadm -> luks -> lvm

With current version udev rules simply having all these disks in the same VG
will result in boot failure, because udev rule will only activate the entire VG
once all PVs are there.

The fix is to use behaviour like in old version and similar to cryptsetup hooks.
Actaully parse what is needed from kernel arg, fstab, etc and try to activate
these LVs directly even if the VG is incomplete. Concretely in my case the hook
should activate verm-r4e/swap (resume device) and verm-r4e/root (root).

This is quite a key regression for servers, as I'm not sure you can access
initramfs shell remotely when it drops into it. Maybe if you have
dropbear-initramfs setup? In my case it's on a workstation.

Could you share thoughts and/or investigate this problem?
Perhaps upstream udev rules are not suited for initramfs and only for regular
system operation once fully booted?

Thanks,

Melvin Vermeeren.

- -- System Information:
Debian Release: bookworm/sid
APT prefers testing
APT policy: (500, 'testing')
merged-usr: no
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 5.18.0-4-amd64 (SMP w/16 CPU threads; PREEMPT)
Kernel taint flags: TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8), LANGUAGE=en_GB:en
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages lvm2 depends on:
ii dmeventd 2:1.02.185-1
ii dmsetup 2:1.02.185-1
ii init-system-helpers 1.64
ii libaio1 0.3.113-2
ii libblkid1 2.38.1-1
ii libc6 2.34-4
ii libdevmapper-event1.02.1 2:1.02.185-1
ii libedit2 3.1-20210910-1
ii libselinux1 3.4-1+b1
ii libsystemd0 251.3-1
ii libudev1 251.3-1
ii lsb-base 11.2
ii systemd [systemd-tmpfiles] 251.3-1

Versions of packages lvm2 recommends:
ii thin-provisioning-tools 0.9.0-2

lvm2 suggests no packages.

- -- Configuration Files:
/etc/lvm/lvm.conf changed [not included]
Edit: I set issue_discards = 1 only.

- -- no debconf information

-----BEGIN PGP SIGNATURE-----

iQJIBAEBCgAyFiEEiu1YAh/qzdXye6Dmpy9idxbqnZYFAmMM7koUHHZlcm1lZXJl
bkB2ZXJtd2EucmUACgkQpy9idxbqnZbDvBAAtOAjlLnoQv6ashtTpuk0b/6MG3UW
BE/we/Z15SC2bsM59zpci9BMsFmfNvS5iVznDC0TRy9P0zaLS+jQ4T4NNd6KPLwe
5mRorEFMI7INwD8vWBkhhCzCXyU6Wp8jCOkngVMND04wU/zq56amKHFnQ9yGaqFl
/jLzTVnUQ2hxJNRvmbghX5x7n4SQw1oSNlBAG8i1Pz2mdMCH2cLWz+ieP2fCYffp
eHkFs+ECTkcnEvDIlRZuIx34+ZZAh2JI/2bPqCtQRZOWOUEujlQDlgEhgp0qORdU
wY2/q1dTsPOwFhgUUvgCtSS0o451E7boZ82EigaGFO/YXWvjiC0eS+rCXokQBJx+
cU89fjgnT/83WlDYN26TiVulX+2peXomZ5c4aCFHQ1jZTbsYePd0ucg2YbwzNAq7
HEQ0hQR6WWVziiHZfrYceyoKgKdcOIpjrnbPra1EHtKGL6o0J5LSzK8ZF45oyD6T
v+gQnzhSEUPL6NPrL+hUSpCwDx70ubLdnNP7n4eiCbaPmTwPoW4p5ohv9Rkeueuz
Ee64BhmVhG3vy8jRHH2gzEmH56oafHfbH7N3tpQtqzuQwdJ6vHs0rniKJaGId6st
9412vuoA3CmmrFFGyha+1gZZif65L6AvXhrqMbcUGUJVuIc20liqAnyDsKEngKJ7
JoJW3tmk89N4uOY=
=8+2U
-----END PGP SIGNATURE-----

[Martin von Gagern]

Similar here. Had been getting error messages about missing PV during boot for a while, but since everything worked fine once booted I didn't bother to resolve that. Now I started getting boot errors instead. Turns out the initramfs was missing a module (pata_atiixp) needed for the controller for that PV. Working this out required time, and until that point I could only boot previous kernel & initramfs (which required navigating grub menu so no headless boots) and I didn't dare upgrade anything.

https://salsa.debian.org/lvm-team/lvm2/-/commit/2340adad4b3875331be1ba7abba881cc1b6e6738 for 2.03.15-1 appears to be the relevant commit here causing this change in behavior. Probably has something to do with the --autoactivation flag which I don't see documented on https://manpages.debian.org/testing/lvm2/vgchange.8.en.html. It probably makes sense for udev to wait for all PVs before activation *if* the PV can be expected to turn up eventually. During boot, falling back to degraded activation after some timeout might be beneficial, in particular to resolve situations where the extra PVs need later boot steps to become available.

The man page describes --activationmode but according to comments in my (unedited) /etc/lvm/lvm.conf that setting is "degraded" by default and should thus be able to work with absent PVs. Particularly since I currently don't even have any LVs on the absent PV. But perhaps --autoactivation trumps that default? If you would like to get more data on why things behave differently, I can probably restore a reproducing setup easily enough and can then execute commands in the initramfs console to debug further.

Steps that helped me fix the problem:

1. "lvm pvs" from the initramfs fallback prompt to see the missing PV

2. "lvm pvs" when booted from previous initramfs to see what device that is, sdc1 in my case

3. "udevadm info -a -n /dev/sdc1 | grep -E 'looking|DRIVER'" to find the driver needed for it

4. "$EDITOR /etc/initramfs-tools/modules" to add the module there

5. "update-initramfs -u -k 5.18.0-4-amd64" to build a new initramfs with that module included

[Guilhem Moulin]

Control: severity -1 serious
Control: tag -1 + patch

Please consider the enclosed patch. The aim is to also activate
incomplete VGs at early boot stage, like lvm2 used to do before
2.03.15-1, and be a no op on “normal systems” once execution has been
handed over to init(1).

There might be a better way to detect an initramfs-tools environment
than checking whether ‘/run/initramfs’ exists (and ‘/run/systemd/system’
doesn't).

The patch doesn't break src:cryptsetup's autopkgtests. It also solves
the present regression AFAICT, at least for the reproducers I tested.

--
Guilhem.

[Christoph Anton Mitterer]

Hey Guilhem.

> There might be a better way to detect an initramfs-tools environment

I once faced the same question when writing a (cryptsetup) keyscript,
i.e. how to definitely find out whether one's "within" the initramfs.


With crypsetup it may seem easy - check for e.g. /scripts/local-
top/cryptroot - but that, as well as similar files (like
/conf/initramfs.conf) may in principle also exist after the system has
booted.

The best I could find was checking whether / is of type rootfs:

Something like:

# `grep`’s `-q`-option is not used as it may cause an exit status of `0` even
# when an error occurred.
grep -E '^[^ ]+ [^ ]+ [^ ]+ [^ ]+ / [^ ]+ ([^ ]+ )*- rootfs [^ ]+ [^ ]+$' /proc/self/mountinfo >/dev/null 2>/dev/null; [ "$?" -ne 1 ];



But checking as you do may in practise fully suffice.


Cheers,
Chris.

[Bastian Blank]

On Wed, May 10, 2023 at 03:09:04AM +0200, Guilhem Moulin wrote:
> There might be a better way to detect an initramfs-tools environment
> than checking whether ‘/run/initramfs’ exists (and ‘/run/systemd/system’
> doesn't).

The only acceptable way would be a different udev rules file for the
initramfs.

Then, degraded is the default activation mode, so overriding that would
not be appropriate. But forcefully enabling stuff like an incomplete
raid will trigger rebuilds every time. So no, this can't be the default
option.

Bastian

--
Conquest is easy. Control is not.
-- Kirk, "Mirror, Mirror", stardate unknown

[Guilhem Moulin]

On Thu, 11 May 2023 at 18:12:52 +0200, Bastian Blank wrote:
> Nope, not really. Half VG was never a real thing. It might work in
> some cases.

And these use-cases are unbootable since 2.03.15…

> Then, degraded is the default activation mode, so overriding that would
> not be appropriate. But forcefully enabling stuff like an incomplete
> raid will trigger rebuilds every time. So no, this can't be the default
> option.

If that's a concern then ‘--activationmode complete’ can be used instead
(although the boot scripts used the default mode from lvm.conf). That's
enough to fix the systems rapported here, because the required LVs are
actually complete.

--
Guilhem.

[Bastian Blank]

On Thu, May 11, 2023 at 06:25:00PM +0200, Guilhem Moulin wrote:
> On Thu, 11 May 2023 at 18:12:52 +0200, Bastian Blank wrote:
> > Nope, not really. Half VG was never a real thing. It might work in
> > some cases.
> And these use-cases are unbootable since 2.03.15…

Those still have access to an old kernel and initramfs, so can be fixed
without special knowledge. The fix is then to use vgsplit to have
complete VG available.

Why it worked in the past: there was a block activation script that just
looked into the "root" parameter and enabled exactly that LV. That was
a fallback to the udev activation.

Bastian

--
You can't evaluate a man by logic alone.
-- McCoy, "I, Mudd", stardate 4513.3

[Melvin Vermeeren]

To add to the previous briefly, as I didn't explicitly mention this.

Early initramfs should:

* Activate the resume device (one of the swap) for resume handling.
* Activate the rootfs.

Once rootfs is there, remaining PVs etc will cascade activate through the
setup from late-stage boot process onwards.

Early initramfs should NOT call vgchange and activate things it doesn't need
to activate, specific lvchange is better in my opinion.

Thanks,

--
Melvin Vermeeren
Systems engineer

[Pásztor János]

Dear Bastian,

On 2023-05-11 17:42, Bastian Blank wrote:
> Control: tags -1 wontfix

>
> On Wed, May 10, 2023 at 03:09:04AM +0200, Guilhem Moulin wrote:

>> Please consider the enclosed patch. The aim is to also activate
>> incomplete VGs at early boot stage, like lvm2 used to do before
>> 2.03.15-1, and be a no op on “normal systems” once execution has been
>> handed over to init(1).

> Nope, not really. Half VG was never a real thing. It might work in
> some cases.

Well, it was a thing between 2014 and 2023, as based on my logs my
machine has that config since then.

>> The patch doesn't break src:cryptsetup's autopkgtests. It also solves
>> the present regression AFAICT, at least for the reproducers I tested.

> I'm actually closing that report. Because half configs are hard to
> handle.
>
> Bastian
>

I would kindly ask you to reconsider this and give a second chance to
similar configs.
Or at least add an entry to here:
https://www.debian.org/releases/testing/amd64/release-notes/ch-information.en.html
so that people could avoid a boot error right after the upgrade.

Regards,
János Pásztor

[Javier Miqueleiz (ethereal)]

I would like to share some more info. Yesterday I had a look at how dracut (version 059-4) manages LVM activation. It uses the older LVM way that allows partial VG activation:

-----------------------------------------

mkdir /var/tmp/dracut

cd /var/tmp/dracut/

lsinitrd --unpack /boot/initrd.img-6.1.0-9-amd64

cat etc/udev/rules.d/64-lvm.rules

...

RUN+="/sbin/initqueue --settled --onetime --unique /sbin/lvm_scan"
RUN+="/sbin/initqueue --timeout --name 51-lvm_scan --onetime --unique /sbin/lvm_scan --activationmode degraded"

...

-----------------------------------------

On one AlmaLinux 9 VM I have for tests, the exact same udev rules are present for the initramfs, so it seems dracut-based distros still support partial VG activation (or at least some of them do).

I've done some initial tests to check if those partial VGs can actually be activated by dracut. The preliminary answer is yes, they can. But I intend to do further tests to check there are no issues with complex storage architectures that involve a combination of RAID, LVM, LUKS, etc.

If reopening this bug and allowing for initramfs-tools to use partial VG activation seems inappropriate, maybe dracut could be a suitable alternative for the users affected by this issue.

Best wishes.

-- 
---- Javier Miqueleiz (ethereal) <javier (at) miqueleiz (dot) com> ----------

  "Since the best man could not be obtained, mediocre ones would have
    to be accepted."

		-- Leipzig mayor Abraham Platz, 1723, commenting on appointing
		   Bach as the Cantor of St Thomas School, Leipzig, when
		   Graupner refused the post (Graupner is a now long-forgotten
		   minor musician); quoted in Werner Neuman, Bach (1961)

--------------------------------------------------------------------------------