Trouble booting lvm raid/luks system with latest kernel in Testing

[Alex Gould]

Hi, I'm hoping for some advice for a problem with my system that is tracking Debian Testing.

The system has two equally-sized internal hard drives. They are partitioned in the following way:

Disk A has a smaller ext2 partition that serves as /boot, with the remainder being a physical volume for LVM

Disk B has one partition, a physical volume for LVM.

The two physical volumes are combined via LVM into one logical volume.

This logical volume is encrypted with cryptsetup using a passphrase (to be entered at boot).

The resulting encrypted volume is then used as a physical volume for LVM, which is divided into two logical volumes: swap and / (root).

This setup was all created via the Debian installer.

Until now, everything seemed to work flawlessly -- Grub loaded the initramfs, I was prompted for the password for the encrypted volume, and then booting proceeded normally to a usable system.

After upgrading to the latest kernel 4.16.0-2-amd64, this no longer worked. Grub loaded, and I got the error message "Cryptsetup: waiting for encrypted source device UUID=365b88eb-195f-43a8-8776-8969ed47744c..." followed a few minutes later by dropping into the initramfs shell.

Boot still works normally if I select the previous kernel, 4.16.0-1-amd64 from the Grub menu.

I imagine I will need to fix some settings for initramfs, grub, crypttab, fstab, or something, but I'm not sure how to proceed.

Any advice on this complicated question is greatly appreciated. Thank you!

[deloptes]

Alex Gould wrote:

> I imagine I will need to fix some settings for initramfs, grub, crypttab,
> fstab, or something, but I'm not sure how to proceed.

I would compare content of new and old initrd

lsinitramfs /boot/initrd-.... > initrd.1.content
lsinitramfs /boot/initrd-.... > initrd.2.content

diff -ub initrd.2.content initrd.1.content

is there something missing?

There are good howtos around, you can try.

check this first
https://wiki.debian.org/InitramfsDebug
https://wiki.debian.org/initramfs -> How to inspect initramfs

and to get the content out

unmkinitramfs /boot/initrd.img-$(uname -r) initramfs/

regards

[Alex Gould]

On Wed, Jul 4, 2018, at 3:08 PM, deloptes wrote:
> Alex Gould wrote:
>
> > I imagine I will need to fix some settings for initramfs, grub, crypttab,
> > fstab, or something, but I'm not sure how to proceed.
>
> I would compare content of new and old initrd
>
> lsinitramfs /boot/initrd-.... > initrd.1.content
> lsinitramfs /boot/initrd-.... > initrd.2.content
>
> diff -ub initrd.2.content initrd.1.content
>
> is there something missing?

Thank you,

There were some differences that might be relevant, unfortunately I'm not sure what they mean. The first change listed below looks suspicious to me.

--- initrd.old.content 2018-07-04 15:10:50.031303390 -0700
+++ initrd.new.content 2018-07-04 15:11:08.463538144 -0700
@@ -28,7 +28,7 @@
conf
conf/conf.d
conf/conf.d/resume
-conf/conf.d/cryptroot
+conf/modules
conf/arch.conf
conf/initramfs.conf
etc

[....]

@@ -1038,10 +1038,10 @@
lib/x86_64-linux-gnu/libm.so.6
lib/x86_64-linux-gnu/libm-2.27.so
lib/x86_64-linux-gnu/libcryptsetup.so.12
-lib/x86_64-linux-gnu/libcryptsetup.so.12.2.0
+lib/x86_64-linux-gnu/libcryptsetup.so.12.3.0
lib/x86_64-linux-gnu/libdevmapper.so.1.02.1
lib/x86_64-linux-gnu/libgcrypt.so.20
-lib/x86_64-linux-gnu/libgcrypt.so.20.2.2
+lib/x86_64-linux-gnu/libgcrypt.so.20.2.3
lib/x86_64-linux-gnu/librt.so.1
lib/x86_64-linux-gnu/librt-2.27.so
lib/x86_64-linux-gnu/libjson-c.so.3
@@ -1077,6 +1077,7 @@
lib/modprobe.d/systemd.conf
lib/cryptsetup
lib/cryptsetup/askpass
+lib/cryptsetup/functions
lib/udev
lib/udev/rules.d
lib/udev/rules.d/56-lvm.rules
@@ -1389,6 +1390,8 @@
init
lib64
lib64/ld-linux-x86-64.so.2
+cryptroot
+cryptroot/crypttab
usr
usr/lib
usr/lib/x86_64-linux-gnu

[tv.d...@googlemail.com]

Hi, I had the same problem on several systems running Sid (unstable), so
it may or may not apply to your case. In my cases I had problems with
systems using a root partition on Luks and also on luks+lvm. When
rebuilding the initramfs I was getting errors (from "update-initramfs")
about the creation of a temporary file. Removing the "cryptroot" and
also on lvm systems the "lvm2" scripts from /etc/initramfs-tools/hooks/
and rebuilding initramfs solved the problem.

Before that I tried various versions of UUID/addressing of partitions in
/etc/crypttab and fstab with no results, tried going through all the
scripts in /etc/initramfs-tools/scripts (and copy fresh ones from
/usr/share/initramfs-tools/), but all this in pure waste of time.
Removing the hook scripts did the trick.

Seems like initramfs luks support is going though a transition, I have
been getting messages about the option "cryptsetup=y" in
"initramfs-tools" being deprecated for a while now, so was expecting
something like this.

Hope it helps. Keep a recovery system from which to boot and chroot
close-by...

[deloptes]

Alex Gould wrote:

> On Wed, Jul 4, 2018, at 3:08 PM, deloptes wrote:
>> Alex Gould wrote:
>>
>> > I imagine I will need to fix some settings for initramfs, grub,
>> > crypttab, fstab, or something, but I'm not sure how to proceed.
>>
>> I would compare content of new and old initrd
>>
>> lsinitramfs /boot/initrd-.... > initrd.1.content
>> lsinitramfs /boot/initrd-.... > initrd.2.content
>>
>> diff -ub initrd.2.content initrd.1.content
>>
>> is there something missing?
>
> Thank you,
>
> There were some differences that might be relevant, unfortunately I'm not
> sure what they mean. The first change listed below looks suspicious to me.
>

Extract the initrd and inspect the '+' new files

> +conf/modules
> +lib/cryptsetup/functions
> +cryptroot/crypttab

finally you have to fix initramfs-tools, to be sure it will create proper
image next time - perhaps consider the answer by tv.d...@googlemail.com
<tv.d...@googlemail.com>

I stopped using systemd as init process long time ago, though keeping it on
the system still creates problems.
As tv.d...@googlemail.com <tv.d...@googlemail.com> reported, it might
be the changes to initramfs-tools.
I do use the stable version to avoid waste of time in such situations,
however I had many times issues with initrd and what I usually do to debug
and solve is to pass to the kernel boot the option init=/bin/sh. When I
enter in the primitive shell I found out what is the problem as I can
manually decrypt, activate lvm and mount root. After this I cd to the mount
point of root and do following (this 3 lines are worth gold)

exec /usr/sbin/chroot . /bin/sh <<- EOF >dev/console 2>&1
exec /sbin/init ${CMDLINE}
EOF

usually this was necessary when swapping drives, so at the end when I boot I
(re)create initrd

Few times a command or two were missing from the initrd, or devices were not
created etc.

regards

[Alex Gould]

Thanks for everyone's help. I looked into what deloptes and tv.debian suggested. In the course of this I have become convinced that I am afflicted by bug #902943 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=902943 .

So I'll subscribe to the bug and see if there's any way I can help troubleshoot or test it.