Bug#964906: initramfs can't boot a raid1 btrfs root volume

[Jason Michaelson]

Package: initramfs-tools

Verston 0.137

I just installed a system using the testing netinst cd image [Debian GNU/Linux bullseye-DI-alpha2 _Bullseye_ - Official Snapshot amd64 NETINST 20200315-11:07]

After install, I converted my btrfs root partition to a raid1 profile, with the understanding that the Debian installer doesn't support that out of the box yet. The problem I've had is that while the initramfs does a btrfs device scan before trying to mount the root file system, when the mount command in the initramfs executes, the second device in the btrfs filesystem goes unfound. 

If I run a btrfs device scan from the initramfs shell i can mount the root file system just fine and continue to boot into the installed operating system.

If i add the script lines from the pre-mount btrfs scripts to the local_mount_root() in  /usr/share/initramfs-tools/scripts/local right before the mount command is executed the initramfs works correctly. I have a hunch this isn't the right solution, despite the fact that it works and I know that the btrfs device scan in the pre-mount btrfs script does in fact get executed. I'm simply unsure what the correct solution actually is with this.

if [ -x /bin/btrfs ]
then
        modprobe btrfs ||:
        # If asked to be quiet, show only errors
        [ "${quiet}" = "y" ] && exec >/dev/null
        /bin/btrfs device scan
fi

[Raphaël Plasson]

I have the same problem on two of my servers (both with similar
configuration, debian buster with backports), both with a btrfs raid-1
on /root.

In my case, initramfs indeed fails to mount /root, but performing a
'btrfs device scan' followed by a manual mount to /root directly on the
initramfs prompts solves the problem.

However, this problem is not systematically met. Even on the same
server, the root may be correctly mounted after a reboot.

Maybe there is a race condition, so that the
'/usr/share/initramfs-tools/scripts/local-premount/btrfs' may not be
launched at the right moment?

Hoping this information can help.

[Michal Sojka]

I experience this after upgrading to bullseye too. Based on my
investigations, the problem is a race between invocation of `btrfs
device scan` in initramfs and kernel block device detection. The scan
is run at the time, when only one of the devices needed for RAID1 has
been detected by the kernel. It appears that initramfs waits for the
root device to be detected by the kernel in `local_device_setup` but
this function is not aware that it needs to wait for both RAID1
devices. Specifically, I have `root=UUID=xxxx` on the kernel command
line and initramfs translates this to /dev/sdb2 via blkid. Initramfs
is not aware of the other needed device /dev/sda2.

If this happens to anybody, you can boot into your system as follows:
Boot with the following on the kernel command line:

break=premount

When the initramfs shell prompt appears wait a second and exit. By
that time all block devices will be detected and boot will proceed
normally.

To ensure that my system boots without manual intervention, I applied
the following workaround. I added
/etc/initramfs-tools/scripts/local-top/btrfs-raid1-wait file containing:

#!/bin/sh

PREREQ=""
prereqs()
{
echo "$PREREQ"
}
case $1 in
prereqs)
prereqs
exit 0
;;
esac
. /scripts/functions

while ! test -b /dev/sda2 || ! test -b /dev/sdb2; do
echo "Waiting for BTRFS RAID1 devices"
sleep 1
done
echo "Done"

Then the initramfs needs to be updated:

update-initramfs -k 5.10.0-8-amd64 -u

[Jason Michaelson]

After two years i did a dist-upgrade on one particular system and this is still an issue. I have to wonder why it got moved to btrfs-progs. This really seems like a sequencing matter because all of the btrfs devices aren't found before the system attempts to mount /. That makes me think that it is an issue with initamfs-tools, especially since those of us who have submitted potential fixes for it in this bug did so against initramfs scripts.

I'm glad I had my script changes elsewhere and could migrate them from one system to another when i encountered this problem. It would just be nice to have it incorporated at the source.

[Matthijs van Duin]

I ran into this same problem. The workarounds suggested here seemed
really hacky so I tried to dig a bit deeper for a proper solution...

I found the "btrfs device ready $DEV" command which is equivalent to
"btrfs device scan $DEV" except it only has exit code 0 if all devices
that are part of the same filesystem are known to the kernel, implying
it is ready to be mounted. Note that contrary to what the btrfs-device
manpage and "btrfs device --help" say it does not actually wait for
this condition, it merely checks and exits.

I also found that udev has a "btrfs ready" builtin which is used by
/lib/udev/rules.d/64-btrfs.rules to set the ID_BTRFS_READY property
based on this same criterion, and it also sets SYSTEMD_READY=0 until
the filesystem is ready, which causes systemd to pretend the device
doesn't exist yet.

I tried testing the ID_BTRFS_READY property in an initramfs script but
for some reason it doesn't seem to get set, even though udev should be
running if I understand correctly. I didn't get around to try using
"btrfs device ready" since I found a much easier workaround:

apt-get install dracut
apt-get purge initramfs-tools-core

Matthijs van Duin

[Kormvel Akos]

There is an easy solution, put rootdelay=5 into the cmdline.txt, this makes init wait before mounting rootfs for 5 seconds and btrfs raid1 starts up with on usb without any issue.