Bug#1052290: cryptsetup-initramfs: askpass is not executed; cryptroot-unlock fails

[Tj]

Package: cryptsetup-initramfs
Version: 2:2.6.1-4~deb12u1
Severity: important

Discovered this whilst working on a relatively simple test of multiple
LUKS block devices for LUKS.0 + LUKS.1 > btrfs RAID1 @/ - that is a
BTRFS RAID1 using 2 LUKS block devices.

Two files represent SSD1 and SSD2, which each have GPT with:

1: EFI-SP (ef00)
2: LUKS (8309) for BTRFS
3: LUKS (8309) for swap

added as loop devices and configured. SSD2's EFI-SP partition is not
formatted.
# fallocate -l 12G ssd${x}.raw
# sgdisk --new=... --typecode=... ssd${x}.raw
# losetup --show --partscan --find ssd${x}.raw
mkfs.vfat -F 16 ${SSD1}p1
# next 2 also applied to SSD2
cryptsetup luksFormat --pbkdf pbkdf2 ${SSD1}p2
cryptsetup open ${SSD1}p2 luks-$(UUID_SSD1p2}
mkfs.btrfs -d raid1 -m raid1 /dev/mapper/luks-${UUID_SSD1p2}
/dev/mapper/luks-${UUID_SSD2p2}
mount /dev/mapper/luks-${UUID_SSD1p2} /target
btrfs subvol create /target/@
btrfs subvol create /target/@home
umount /target
mount -o subvol=@ /dev/mapperluks-${UUID_SSD1p2}
debootstrap bookworm /target
# add and configure packages for bootable EFI image

After unmounting and closing devices create a libvirt VM guest using the
two files as virtio storage and configure for UEFI boot.

On startup GRUB correctly opens the LUKS block devices to access vmlinuz
and initrd.img, and its own configuration and modules.

On reaching initialramfs it fails to unlock either of the LUKS devices;
eventually dropping to the shell after reporting:

Error: Timeout reached while waiting for askpass.

After using `break=mount` and investigating with `sh -x
/bin/cryptsetup-unlock` it seems it fails because it is not finding
`askpass` in the process list.

On closer examination and searching I am unable to locate where
/usr/lib/cryptsetup/askpass is actually executed. `cryptsetup-unlock`
correctly locates the file with [ -f ] and ensures it is executable with
[-x ] but I do not see any attempt to actually execute it.

If needed I can either share the 2 SSD files or a script to build them.

-- System Information:
Debian Release: 12.1
Architecture: amd64 (x86_64)

Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8), LANGUAGE=en_GB:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages cryptsetup-initramfs depends on:
ii busybox-static [busybox] 1:1.36.0-1~exp1
ii cryptsetup 2:2.6.1-4~deb12u1
ii debconf [debconf-2.0] 1.5.82
ii initramfs-tools [linux-initramfs-tool] 0.143~tj01

Versions of packages cryptsetup-initramfs recommends:
ii console-setup 1.221
ii kbd 2.5.1-1+b1

[Guilhem Moulin]

Control: tag -1 moreinfo

On Tue, 19 Sep 2023 at 22:39:40 +0100, Tj wrote:
> On reaching initialramfs it fails to unlock either of the LUKS devices;
> eventually dropping to the shell after reporting:
>
> Error: Timeout reached while waiting for askpass.
>
> After using `break=mount` and investigating with `sh -x
> /bin/cryptsetup-unlock` it seems it fails because it is not finding
> `askpass` in the process list.

cryptsetup-unlock waits until the initramfs boot script is blocking on
passphrase prompt. This is only useful for injecting passphrases from
outside the initramfs console (for instance from a remote shell).

When you set ‘break=mount’ our boot script isn't running in the
background, so cryptsetup-unlock timeouts. This is expected. Why are
you running cryptsetup-unlock in the first place instead of relying on
the builtin initramfs logic? Also, FWIW ‘break=mount’ breaks *after*
unlocking whatever block devices need to be unlocked, so cryptsetup-initramfs
has nothing more to do at this point.

--
Guilhem.

[Tj]

Apologies for a mis-type there; I am using `break=premount`.

The install "appears" to fail to find and unlock both LUKS devices;
it "hangs" (pauses) for a very long time with only kernel messages showing but
no passphrase prompt is shown.

After debugging it turns out the kernel command-line option `debug` and initramfs-tools
`init` script cause the passphrase prompts to be redirected to `initramfs.debug` when askpass
is using "console" output.

It might be better if askpass writes the prompt directly to `/dev/console`. I've done a basic
test and this does work when stdin/stderr are redirected. See end of this email.

Tapping `[Enter]` a few times will sometimes cause a few blank lines to
scroll and seem to move things on since a few more kernel messages are
generated (showing that more udev rules are being triggered).

Eventually, after more taps of `[Enter]` (or maybe due to a coincidental timeout)
it does drop to shell. This is after something like 600 seconds (10 minutes).

Then I find:

REASON=ALERT! UUID=cb2a... does not exist. Dropping to shell!

`/proc/cmdline`: BOOT_IMAGE=/@/boot/vmlinuz... root=UUID=cb2a... ro rootflags=subvol=@ debug systemd.log_level=info

Looking at `/run/initramfs/initramfs.debug` today I see the reason for it failing:
when using `debug` the init script does:
```
debug)
debug=y
quiet=n
if [ -n "${netconsole}" ]; then
log_output=/dev/kmsg
else
log_output=/run/initramfs/initramfs.debug
fi
set -x
;;
```
And looking at `/run/initramfs/initramfs.debug` it has 'captured' the passphrase prompts:
```
+ /scripts/local-top/cryptroot
Please unlock disk luks-88faf2e0-400b-4533-bdab-11ddb8d07fa7:
Nothing to read on input.
cryptsetup: ERROR: luks-88faf2e0-400b-4533-bdab-11ddb8d07fa7: cryptsetup
failed, bad password or options?
Please unlock disk luks-88faf2e0-400b-4533-bdab-11ddb8d07fa7:
Nothing to read on input.
cryptsetup: ERROR: luks-88faf2e0-400b-4533-bdab-11ddb8d07fa7: cryptsetup
failed, bad password or options?
Please unlock disk luks-88faf2e0-400b-4533-bdab-11ddb8d07fa7:
Nothing to read on input.
cryptsetup: ERROR: luks-88faf2e0-400b-4533 if (fwrite(prompt_ptr, line_len, 1, stderr) < 1 ||
fwrite("\n", 1, 1, stderr) < 1) {
-bdab-11ddb8d07fa7: cryptsetup
failed, bad password or options?
cryptsetup: ERROR: luks-88faf2e0-400b-4533-bdab-11ddb8d07fa7: maximum number of
tries exceeded
```
Typing the passphrases blind when the kernel messages 'hang' with a decent pause between them
to allow for the time it takes to 'unlock' allows one to continue starting successfully.

So the issue isn't askpass not running but the lack of a passphrase prompt when using
`debug` on kernel command line AND askpass using its "console" method.
I set `debug` by default on all hosts but on most that use cryptsetup-initramfs the host
is using key-files so never prompts for a passphrase and therefore I never hit this issue before.

Diagnosing why the prompt is 'lost' `run_keyscript()` does:
```
elif [ "$keyscriptarg" = "none" ]; then
# don't export the prompt message as CRYPTTAB_KEY
keyscript="/lib/cryptsetup/askpass"
keyscriptarg="Please unlock disk $CRYPTTAB_NAME: "
fi

exec "$keyscript" "$keyscriptarg"
```
and that is the source of the prompt (there are 2 places where an identical prompt
is set/written).

I enabled DEBUG in `askpass` and copied it into the initrd to capture `/tmp/askpass.debug`
and then allowed init to continue (`exit`).

I had to tap [Enter] a few times to get back to the shell, then looking at the log:
```
Enabling method systemd
Enabling method fifo
Enabling method plymouth
Enabling method console
method 1 has fd 4 and name fifo
method 3 has fd 0 and name console
Starting select with nfds 5
Writing 0 bytes to stdout
Disabling method ALL
```
This is repeated six times (presumably three tries each for the two LUKS devices).

So when using its "console" method stdout is redirected by initramfs-tools `init` when
`debug` is set:
```
if [ -n "$log_output" ]; then
exec >"$log_output" 2>&1
unset log_output
fi
```
and as askpass.c::console_prepare() does:
```
if (fwrite(prompt_ptr, line_len, 1, stderr) < 1 ||
fwrite("\n", 1, 1, stderr) < 1) {
```
the output is swallowed.

I wrote a proof-of-concept to write to `/dev/console` to avoid the redirects and it
appears to work successfully in the initialramfs:
```
(initramfs) /mnt/usb/test_console >/tmp/test.log 2>&1
Testing /dev/console
```
Code:
```
#include <stdio.h>
#include <unistd.h>

int main(int argc, char **argv, char **env)
{
char *console = "/dev/console";
if (access(console, W_OK) < 0) {
perror("access('/dev/console') ");
return -1;
}

FILE *out = fopen(console, "a");
if (!out) {
fprintf(stderr, "Failed to open %s\n", console);
return -2;
} else {
fprintf(out, "\nTesting %s\n", console);
fclose(out);
}
return 0;
}
```
So I wonder if this would be possible with askpass?