Bug#838001: cryptsetup does not support ZFS

[Richard Laager]

Package: cryptsetup

I use Ubuntu, not Debian, and filed this bug:
https://bugs.launchpad.net/ubuntu/+source/cryptsetup/+bug/1612906

It was suggested there that I look at Debian bug #820888, titled
cryptsetup: initramfs cryptroot zfs support:
https://bugs.debian.org/820888

I think that's a separate issue, but I'm not 100% sure. Either way, it'd
be nice to upstream the patch below into Debian's cryptsetup, if the
patch is in fact suitable for Debian.

On to the actual issue...

This happens on Ubuntu 16.04:
$ sudo update-initramfs -c -k all
update-initramfs: Generating /boot/initrd.img-4.4.0-34-generic
cryptsetup: WARNING: could not determine root device from /etc/fstab
update-initramfs: Generating /boot/initrd.img-4.4.0-31-generic
cryptsetup: WARNING: could not determine root device from /etc/fstab

The attached patch adds ZFS support to cryptsetup.

--
Richard

[Guilhem Moulin]

Hi Richard,

On Fri, 16 Sep 2016 at 04:32:24 -0500, Richard Laager wrote:
> The attached patch adds ZFS support to cryptsetup.

Thanks the patch. Unfortunately I can't test it easily, and I know
nothing about ZFS, but I'll try to review your patch anyway.

> + zfs list -H -o name,canmount,mountpoint 2>/dev/null | \

Looks like there is support for ZFS filesystems in the /etc/fstab
<https://wiki.freebsd.org/RootOnZFS#Alternate_.2Fetc.2Ffstab>. Since
it's what other FS are using, I think that's what we should use for ZFS
too. Don't the ‘name’ and ‘mountpoint’ columns of your ‘zfs list’
command respectively correspond to the first and second columns of
fstab(5)?

> + for dev in $(zpool status -P "${name%%/*}" 2>/dev/null | awk '($1 ~ /\//) {print $1}'); do

This looks rather brittle, as the ‘status’ command seems to be intended
for humans not parsers. Isn't there a more robust way to retrieve the
underlying device list of a ZFS pool? (FWIW, our current solution for
BTRFS volumes isn't satisfactory either.)

Cheers,
--
Guilhem.

[Richard Laager]

On 09/17/2016 06:45 PM, Guilhem Moulin wrote:
> On Fri, 16 Sep 2016 at 04:32:24 -0500, Richard Laager wrote:
>> The attached patch adds ZFS support to cryptsetup.
>
> Thanks the patch. Unfortunately I can't test it easily, and I know
> nothing about ZFS, but I'll try to review your patch anyway.

Thanks for taking the time to look at this. I'm happy to help in
whatever way I can. Whatever you need, just ask, and I'll do my best.

You might find the upstream ZFS-on-Linux project's root-on-ZFS HOWTO
(which I maintain) informative, even though it's written for Ubuntu:
https://github.com/zfsonlinux/zfs/wiki/Ubuntu-16.04-Root-on-ZFS

I have changes all ready-to-go for supporting LUKS, but this "bug"
(read: feature request) is a blocker. It is possible to work-around this
with (otherwise unused) /etc/fstab entries, but I don't want to
encourage that in the wild, lest it cause some other problem in the future.

My general approach to using LUKS with ZFS is that GRUB prompts for the
passphrase, unlocks LUKS, which ZFS is inside, and reads the kernel and
initrd. When the kernel boots, the initrd contains a keyfile which is
used to unlock LUKS again. This maintains the goal of having everything
including /boot, but excluding GRUB, on ZFS.

However, I am aware that it deviates from the Ubuntu approach of an
unencrypted /boot with an initrd that prompts for the LUKS passphrase.
So I'm forced to choose between two major goals: 1) everything on ZFS &
2) stay as close to normal Ubuntu as possible.

Here are some selected parts of the diff, which outline how I'm using
LUKS, ZFS, and cryptsetup.

+ # cryptsetup luksFormat -c aes-xts-plain64 -s 256 -h sha256 \
+ /dev/disk/by-id/scsi-SATA_disk1-part1
+ # cryptsetup luksOpen /dev/disk/by-id/scsi-SATA_disk1-part1 luks1
+ # zpool create -o ashift=12 \
+ -O atime=off -O canmount=off -O compression=lz4 -O
normalization=formD \
+ -O mountpoint=/ -R /mnt \
+ rpool /dev/mapper/luks1


+ # mkdir /etc/keys
+ # dd bs=64 count=1 if=/dev/urandom of=/etc/keys/root.key
+ # chmod 600 /etc/keys/root.key
+ # cryptsetup luksAddKey /dev/disk/by-id/scsi-SATA_disk1-part1
/etc/keys/root.key
+ # echo luks1 UUID=$(blkid -s UUID -o value \
+ /dev/disk/by-id/scsi-SATA_disk1-part1) /etc/keys/root.key \
+ luks,discard > /etc/crypttab
+
+ # vi /etc/initramfs-tools/initramfs.conf
+ Add these lines:
+ export KEYFILE_PATTERN="/etc/keys/*.key"
+ UMASK=077

On a side note, it'd be nice if cryptsetup would automatically set UMASK
when a keyfile is found. And it'd be nice if there as a default where it
looked for keyfiles.


As another resource, here's the upstream HOWTO for Debian, which someone
else maintains:
https://github.com/zfsonlinux/zfs/wiki/HOWTO-install-Debian-GNU-Linux-to-a-Native-ZFS-Root-Filesystem

The HOWTOs share some history, but are not developed in lock-step. The
packaging for Debian and Ubuntu has also been different historically.
Now that ZFS has been accepted into both Ubuntu and Debian, people are
working on reducing the differences.

>> + zfs list -H -o name,canmount,mountpoint 2>/dev/null | \
>
> Looks like there is support for ZFS filesystems in the /etc/fstab
> <https://wiki.freebsd.org/RootOnZFS#Alternate_.2Fetc.2Ffstab>. Since
> it's what other FS are using, I think that's what we should use for ZFS
> too.

ZFS does not normally use /etc/fstab. That's not to say it's impossible
(obviously, it is), but that is absolutely not what users expect of ZFS
nor how it is used in practice.

I don't know why that wiki page leans so heavily on mountpoint=legacy
and /etc/fstab. You won't find that in other ZFS documentation for any
OS, including the more up-to-date FreeBSD documentation linked from the
header on that page.

> Don't the ‘name’ and ‘mountpoint’ columns of your ‘zfs list’
> command respectively correspond to the first and second columns of
> fstab(5)?

Yes.

>> + for dev in $(zpool status -P "${name%%/*}" 2>/dev/null | awk '($1 ~ /\//) {print $1}'); do
>
> This looks rather brittle, as the ‘status’ command seems to be intended
> for humans not parsers.

It's not ideal, I admit, but it's "how it's done". The -P flag was added
specifically because the output of `zpool status` is frequently parsed.
Parsing `zpool status` is exactly how GRUB handles ZFS. If its output
changes, GRUB breaks too.

Ideally, there'd be a -H flag that omits extraneous output and spaces,
like on the zfs command.

> Isn't there a more robust way to retrieve the
> underlying device list of a ZFS pool?

Using libzfs is another way, if your application is compiled. However,
that can be a problem for certain applications because of licensing. For
example, GRUB uses libzfs on Solaris where they can take advantage of
the system library exception in the GPL, but they parse `zpool status`
on other OSes, including Linux.

Since cryptsetup is a shell script, a C library is a non-starter anyway.

--
Richard

[Guilhem Moulin]

On Sat, 17 Sep 2016 at 19:37:23 -0500, Richard Laager wrote:
> I have changes all ready-to-go for supporting LUKS, but this "bug"
> (read: feature request) is a blocker. It is possible to work-around this
> with (otherwise unused) /etc/fstab entries, but I don't want to
> encourage that in the wild, lest it cause some other problem in the future.

The fstab(5) entry for the root FS is never required, is it? Mine is
mounted (at initramfs stage) using the “root=” and “rootflags=” kernel
parameters. In theory that should work for ZFS as well, but it seems
like it's currently not the case (cf. #820888). The kernel parameters
approach and the /etc/fstab approach are both FS-agnostic, and I'd
rather not have FS-specific code to find out which device is mounted on
/ or /usr.

> My general approach to using LUKS with ZFS is that GRUB prompts for the
> passphrase, unlocks LUKS, which ZFS is inside, and reads the kernel and
> initrd. When the kernel boots, the initrd contains a keyfile which is
> used to unlock LUKS again. This maintains the goal of having everything
> including /boot, but excluding GRUB, on ZFS.

This is a nice setup, but it should work for other filesystems, too.

> + # vi /etc/initramfs-tools/initramfs.conf
> + Add these lines:
> + export KEYFILE_PATTERN="/etc/keys/*.key"

Setting “KEYFILE_PATTERN” in /etc/initramfs-tools/initramfs.conf will
be deprecated in the next release, cf.

https://anonscm.debian.org/cgit/pkg-cryptsetup/cryptsetup.git/commit/?id=94f7f779944ae7082dac7ce993174a8531b54209

Configuration for the cryptroot initramfs hook script is now to be
placed in /etc/crytsetup-initramfs/conf-hook.

> On a side note, it'd be nice if cryptsetup would automatically set UMASK
> when a keyfile is found. And it'd be nice if there as a default where it
> looked for keyfiles.

Feel free to open new wishlist bugs? :-) Note however that setting
UMASK conditionally seems difficult, since UMASK is a mkinitramfs(8)
option, while KEYFILE_PATTERN is an option of the cryptroot initramfs
hook. My commit from a few days ago makes the hook script print a loud
warning if the value of UMASK is too permissive:

https://anonscm.debian.org/cgit/pkg-cryptsetup/cryptsetup.git/commit/?id=a7a74f27bc44f73b479d79219fe79fe864f40e30

but since it's run as a subprocess, it's not possible to set UMASK in
the hook itself.

>>> + zfs list -H -o name,canmount,mountpoint 2>/dev/null | \
>>
>> Looks like there is support for ZFS filesystems in the /etc/fstab
>> <https://wiki.freebsd.org/RootOnZFS#Alternate_.2Fetc.2Ffstab>. Since
>> it's what other FS are using, I think that's what we should use for ZFS
>> too.
>
> ZFS does not normally use /etc/fstab. That's not to say it's impossible
> (obviously, it is), but that is absolutely not what users expect of ZFS
> nor how it is used in practice.

Fair enough. But I still think we can find a solution that's not
FS-specific. For instance, to ensure that an arbitrary device is
unlocked at initramfs stage, you can add ‘initramfs’ to the 4th column
of its crypttab(5) entry. In your case, that should be enough to copy
the key files of the underlying devices to the initramfs image. Then —
modulo #820888 — the initramfs boot script should be able to decrypt
these devices and mount the root FS.

> I don't know why that wiki page leans so heavily on mountpoint=legacy
> and /etc/fstab. You won't find that in other ZFS documentation for any
> OS, including the more up-to-date FreeBSD documentation linked from the
> header on that page.

Alright, thanks for the clarification. As I said, I'm clueless about
ZFS ;-)

>>> + for dev in $(zpool status -P "${name%%/*}" 2>/dev/null | awk '($1 ~ /\//) {print $1}'); do
>>
>> This looks rather brittle, as the ‘status’ command seems to be intended
>> for humans not parsers.
>
> It's not ideal, I admit, but it's "how it's done". The -P flag was added
> specifically because the output of `zpool status` is frequently parsed.
> Parsing `zpool status` is exactly how GRUB handles ZFS. If its output
> changes, GRUB breaks too.
>
> Ideally, there'd be a -H flag that omits extraneous output and spaces,
> like on the zfs command.

Alright. Can you provide an example of the output produced by `zpool
status -P "${name%%/*}"`, though? I don't know if adding ‘-P’ changes
anything, but at least without it I saw headers containing slash
characters.

>> Isn't there a more robust way to retrieve the
>> underlying device list of a ZFS pool?
>
> Using libzfs is another way, if your application is compiled. However,
> that can be a problem for certain applications because of licensing. For
> example, GRUB uses libzfs on Solaris where they can take advantage of
> the system library exception in the GPL, but they parse `zpool status`
> on other OSes, including Linux.
>
> Since cryptsetup is a shell script, a C library is a non-starter anyway.

Yeah, sure. I was hopping for another ‘zpool’ subcommand.

--
Guilhem.

[Richard Laager]

On 09/18/2016 02:13 PM, Guilhem Moulin wrote:
> On Sat, 17 Sep 2016 at 19:37:23 -0500, Richard Laager wrote:
>> I have changes all ready-to-go for supporting LUKS, but this "bug"
>> (read: feature request) is a blocker. It is possible to work-around this
>> with (otherwise unused) /etc/fstab entries, but I don't want to
>> encourage that in the wild, lest it cause some other problem in the future.
>
> The fstab(5) entry for the root FS is never required, is it?

No, I don't think so.

And ZFS *never* uses /etc/fstab, for the root FS or any FS.

> Mine is
> mounted (at initramfs stage) using the “root=” and “rootflags=” kernel
> parameters. In theory that should work for ZFS as well, but it seems
> like it's currently not the case (cf. #820888).

I looked into 820888 some more. That person is doing ZFS on top of LVM
on top of LUKS. (Or ZFS inside LVM inside LUKS, if you prefer that way
of looking at it.) While that's theoretically something that "should"
work, that's not a very practical configuration. Specifically, using
two volume managers is ill-advised in my opinion. I recommend putting
swap on ZFS instead of that setup.

I completely agree with your analysis that it is not immediately
apparent why cmdline_root is getting set to root=ZFS=..., and that they
should enable debugging to help figure that out.

I commented to this effect on that bug report.

> The kernel parameters
> approach and the /etc/fstab approach are both FS-agnostic, and I'd
> rather not have FS-specific code to find out which device is mounted on
> / or /usr.

When someone is using ZFS, there is simply no way to get from a
mountpoint to a set of devices without invoking ZFS-specific support
code. Looking at /etc/fstab, mount output, or /proc/self/mounts (or
/etc/mtab) will only give you a ZFS dataset name. That is, / will
convert to yakkety/ROOT/ubuntu, not /dev/mapper/luks1. You need to ask
zpool (or libzfs) to get from yakkety/ROOT/ubuntu to /dev/mapper/luks1.

> Configuration for the cryptroot initramfs hook script is now to be
> placed in /etc/crytsetup-initramfs/conf-hook.

Thanks for letting me know. I've made a note and will adjust this in
the HOWTO once this lands in Ubuntu.

> since it's run as a subprocess, it's not possible to set UMASK in
> the hook itself.

Right, this is probably something I'd have to bring up with
initramfs-tools, not cryptsetup. And cryptsetup integration aside, I'm
not entirely sure why it can't use a restrictive umask by default.

> Can you provide an example of the output produced by `zpool
> status -P "${name%%/*}"`, though? I don't know if adding ‘-P’ changes
> anything, but at least without it I saw headers containing slash
> characters.

Where are you seeing headers with slash characters? I can't think of
any scenario where that might occur. And even if it did, it would have
to be the first field to be grabbed by the awk command I wrote.

With -P, zpool status shows the full path to the device. Always use -P
these days, in any script. Without -P, ZFS "wholedisk" pools will show
"sda" when you may be expecting "sda1"; in this case, using -P shows
/dev/sda1, which is what you want.

Also, without -P, you have no idea which directory the device file is
in. It can be in /dev, or it can be in /dev/disk/by-id, or /dev/mapper,
or something else.

You can also add -L to resolve symlinks. (Using -L without -P gives the
basename of the resolved symlink.)

Here is my test system running Ubuntu 16.10 "Yakkety" with LUKS:

root@yakkety:~# zfs list -H -o name,canmount,mountpoint
yakkety off /
yakkety/ROOT off none
yakkety/ROOT/ubuntu noauto /
yakkety/home on /home
yakkety/home/root on /root
yakkety/var off /var
yakkety/var/log on /var/log
yakkety/var/spool on /var/spool
yakkety/var/tmp on /var/tmp

Here's what the loop would do with that:

root@yakkety:~# name=yakkety/ROOT/ubuntu

The variable expansion is to cut that to just the pool name:

root@yakkety:~# echo "${name%%/*}"
yakkety

Full zpool status output:

root@yakkety:~# zpool status -P "${name%%/*}"
pool: yakkety
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(5) for details.
scan: none requested
config:

NAME STATE READ WRITE CKSUM
yakkety ONLINE 0 0 0
/dev/mapper/luks1 ONLINE 0 0 0

errors: No known data errors
root@yakkety:~# zpool status "${name%%/*}"
pool: yakkety
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(5) for details.
scan: none requested
config:

NAME STATE READ WRITE CKSUM
yakkety ONLINE 0 0 0
luks1 ONLINE 0 0 0

errors: No known data errors

root@yakkety:~# zpool status -LP "${name%%/*}"
pool: yakkety
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(5) for details.
scan: none requested
config:

NAME STATE READ WRITE CKSUM
yakkety ONLINE 0 0 0
/dev/dm-0 ONLINE 0 0 0

errors: No known data errors

Final output:

root@yakkety:~# zpool status "${name%%/*}" | awk '($1 ~ /\//) {print $1}'
root@yakkety:~# zpool status -P "${name%%/*}" | awk '($1 ~ /\//) {print $1}'
/dev/mapper/luks1
root@yakkety:~# zpool status -LP "${name%%/*}" | awk '($1 ~ /\//) {print $1}'
/dev/dm-0

--
Richard

[Guilhem Moulin]

On Mon, 19 Sep 2016 at 00:51:45 -0500, Richard Laager wrote:
> On 09/18/2016 02:13 PM, Guilhem Moulin wrote:
>> The kernel parameters
>> approach and the /etc/fstab approach are both FS-agnostic, and I'd
>> rather not have FS-specific code to find out which device is mounted on
>> / or /usr.
>
> When someone is using ZFS, there is simply no way to get from a
> mountpoint to a set of devices without invoking ZFS-specific support
> code. Looking at /etc/fstab, mount output, or /proc/self/mounts (or
> /etc/mtab) will only give you a ZFS dataset name. That is, / will
> convert to yakkety/ROOT/ubuntu, not /dev/mapper/luks1. You need to ask
> zpool (or libzfs) to get from yakkety/ROOT/ubuntu to /dev/mapper/luks1.

Sorry, I didn't use the right terminology. I meant I'd rather avoid
FS-specific code to find out which *pool/dataset* is mounted on a
particular mountpoint. Of course, like for BTRFS (and mdadm if one day
we add support for RAID on top of LUKS, cf. #629236), we would need
FS-specific code to retrieve the list of underlying devices.

>> Can you provide an example of the output produced by `zpool
>> status -P "${name%%/*}"`, though? I don't know if adding ‘-P’ changes
>> anything, but at least without it I saw headers containing slash
>> characters.
>
> Where are you seeing headers with slash characters? I can't think of
> any scenario where that might occur. And even if it did, it would have
> to be the first field to be grabbed by the awk command I wrote.

https://www.freebsd.org/doc/handbook/zfs-zpool.html
https://docs.oracle.com/cd/E23824_01/html/821-1448/gaynp.html#gamno

Fair enough it's not the first field, but by looking at the output I
have the impression that the header is pretty much free form. I don't
see why the URL couldn't end up on the next line after wrapping, for
instance.

> With -P, zpool status shows the full path to the device. Always use -P
> these days, in any script. Without -P, ZFS "wholedisk" pools will show
> "sda" when you may be expecting "sda1"; in this case, using -P shows
> /dev/sda1, which is what you want.
>
> Also, without -P, you have no idea which directory the device file is
> in. It can be in /dev, or it can be in /dev/disk/by-id, or /dev/mapper,
> or something else.

Alright, thanks! I think a more robust AWK or sed script is desired to
strip out the header, though. Perhaps this AWK script

{
while(1) {
i = index($0, ":")
cfg = /^[\t ]*config:/
while(1) {
if (getline != 1)
exit
if (/[^\t ]/ && match($0, /[^\t ]/) <= i)
break
if (cfg && /[^\t ]*\//)
print $1
}
}
}

By the way, what do you think about my suggestion to use the crypttab(5)
to consider the relevant devices for unlocking at initramfs stage?

>> But I still think we can find a solution that's not FS-specific. For
>> instance, to ensure that an arbitrary device is unlocked at initramfs
>> stage, you can add ‘initramfs’ to the 4th column of its crypttab(5)
>> entry. In your case, that should be enough to copy the key files of
>> the underlying devices to the initramfs image. Then — modulo #820888
>> — the initramfs boot script should be able to decrypt these devices
>> and mount the root FS.

Again, I'm not fond of your `zfs list` command because it's an
FS-specific alternative for two FS-agnostic and de facto standard
solutions, namely fstab(5) and kernel parameters.

--
Guilhem.

[Richard Laager]

On 09/19/2016 04:48 AM, Guilhem Moulin wrote:
> On Mon, 19 Sep 2016 at 00:51:45 -0500, Richard Laager wrote:
>> On 09/18/2016 02:13 PM, Guilhem Moulin wrote:

> Fair enough it's not the first field, but by looking at the output I
> have the impression that the header is pretty much free form.

Fair point.

At an absolute minimum, the regex should be tightened up to require $1
to start with / rather than just contain it. Further options below...

> Alright, thanks! I think a more robust AWK or sed script is desired to
> strip out the header, though. Perhaps this AWK script

That awk script doesn't output anything for me.

How about this, which I used in the attached updated patch:
awk 'NR == 1, /^[\t ]*config:/ { next } { if ($1 ~ /^\//) { print $1 } }'

Or if you want to match the logic of GRUB2 (which I think is messier):
awk 'NR == 1, /^[\t ]*NAME[\t ]+STATE[\t ]+READ[\t ]+WRITE[\t ]+CKSUM/ { next } { if ($1 ~ /^\//) { print $1 } }'

>>> But I still think we can find a solution that's not FS-specific. For
>>> instance, to ensure that an arbitrary device is unlocked at initramfs
>>> stage, you can add ‘initramfs’ to the 4th column of its crypttab(5)
>>> entry. In your case, that should be enough to copy the key files of
>>> the underlying devices to the initramfs image.

Yes, that works. The "WARNING: could not determine root device" message
is still printed.

> Again, I'm not fond of your `zfs list` command because it's an
> FS-specific alternative for two FS-agnostic and de facto standard
> solutions, namely fstab(5) and kernel parameters.

I've attached another version of this patch. It first swaps from
/etc/fstab to /proc/self/mounts. Then adding ZFS support is trivial and
integrates in the same way as btrfs support.

--
Richard

[Guilhem Moulin]

On Mon, 19 Sep 2016 at 06:39:35 -0500, Richard Laager wrote:
> On 09/19/2016 04:48 AM, Guilhem Moulin wrote:
>> On Mon, 19 Sep 2016 at 00:51:45 -0500, Richard Laager wrote:
>>> On 09/18/2016 02:13 PM, Guilhem Moulin wrote:
>> Alright, thanks! I think a more robust AWK or sed script is desired to
>> strip out the header, though. Perhaps this AWK script
>
> That awk script doesn't output anything for me.

Interesting, worksforme™ both with gawk(1) and mawk(1). Perhaps an
issue with TAB vs spaces? Could you redirect the output of `zfs status
-P $pool` to a file and attach that file instead of copy-paste the
output?

> How about this, which I used in the attached updated patch:
> awk 'NR == 1, /^[\t ]*config:/ { next } { if ($1 ~ /^\//) { print $1 } }'
>
> Or if you want to match the logic of GRUB2 (which I think is messier):
> awk 'NR == 1, /^[\t ]*NAME[\t ]+STATE[\t ]+READ[\t ]+WRITE[\t ]+CKSUM/ { next } { if ($1 ~ /^\//) { print $1 } }'

Well sorry for the nitpick, but both are broken if the command happens
to produce an output for which triggering line matches the wrapped value
of another header. For instance consider

note: blah blah see under
config: for details.
This is the remaining "note:", not "config:"

That being said, if as you said GRUB would choke on such corner cases
maybe it's not high priority in cryptsetup. But it doesn't hurt to try
to get parsing right either ;-)

>>>> But I still think we can find a solution that's not FS-specific. For
>>>> instance, to ensure that an arbitrary device is unlocked at initramfs
>>>> stage, you can add ‘initramfs’ to the 4th column of its crypttab(5)
>>>> entry. In your case, that should be enough to copy the key files of
>>>> the underlying devices to the initramfs image.
>
> Yes, that works. The "WARNING: could not determine root device" message
> is still printed.

That's just a warning :-) But maybe it should go away when “root=” is
found in /proc/cmdline.

>> Again, I'm not fond of your `zfs list` command because it's an
>> FS-specific alternative for two FS-agnostic and de facto standard
>> solutions, namely fstab(5) and kernel parameters.
>
> I've attached another version of this patch. It first swaps from
> /etc/fstab to /proc/self/mounts. Then adding ZFS support is trivial and
> integrates in the same way as btrfs support.

Hmm, swapping /etc/fstab for /proc/self/mounts seems like a major
change, with potential undesired consequences. Up to now the device
associated to a given mountpoint was the one found in fstab(5), which is
not necessarily the one currently mounted.

--
Guilhem.

[Richard Laager]

On 09/19/2016 07:21 AM, Guilhem Moulin wrote:
> On Mon, 19 Sep 2016 at 06:39:35 -0500, Richard Laager wrote:
>> On 09/19/2016 04:48 AM, Guilhem Moulin wrote:
>>> On Mon, 19 Sep 2016 at 00:51:45 -0500, Richard Laager wrote:
>>>> On 09/18/2016 02:13 PM, Guilhem Moulin wrote:
>>> Alright, thanks! I think a more robust AWK or sed script is desired to
>>> strip out the header, though. Perhaps this AWK script
>>
>> That awk script doesn't output anything for me.
>
> Interesting, worksforme™ both with gawk(1) and mawk(1). Perhaps an
> issue with TAB vs spaces? Could you redirect the output of `zfs status
> -P $pool` to a file and attach that file instead of copy-paste the
> output?

I'm using mawk. File attached.

I'm intentionally trying to keep the parser as simple as possible. I
don't want to introduce bugs by being "too clever". Upstream changes to
the `zpool status` output are likely to be weighed and tested very
carefully.

>>> Again, I'm not fond of your `zfs list` command because it's an
>>> FS-specific alternative for two FS-agnostic and de facto standard
>>> solutions, namely fstab(5) and kernel parameters.
>>
>> I've attached another version of this patch. It first swaps from
>> /etc/fstab to /proc/self/mounts. Then adding ZFS support is trivial and
>> integrates in the same way as btrfs support.
>
> Hmm, swapping /etc/fstab for /proc/self/mounts seems like a major
> change, with potential undesired consequences. Up to now the device
> associated to a given mountpoint was the one found in fstab(5), which is
> not necessarily the one currently mounted.

That's why I didn't do this originally. Changing to /proc/self/mounts is
a simpler/clearer, but more significant/riskier code change.

If you want to maintain the semantics of "what is configured" rather
than "what is mounted", then the only way to support ZFS is to call `zfs
list`. That is how you find out "what is configured" in ZFS.

--
Richard

[Guilhem Moulin]

Control: severity -1 wishlist

On Mon, 19 Sep 2016 at 08:21:13 -0500, Richard Laager wrote:
> On 09/19/2016 07:21 AM, Guilhem Moulin wrote:
>> Interesting, worksforme™ both with gawk(1) and mawk(1). Perhaps an
>> issue with TAB vs spaces? Could you redirect the output of `zfs status
>> -P $pool` to a file and attach that file instead of copy-paste the
>> output?
>
> I'm using mawk. File attached.

I see. My bad, I thought the leading tabs were spaces.

>> Hmm, swapping /etc/fstab for /proc/self/mounts seems like a major
>> change, with potential undesired consequences. Up to now the device
>> associated to a given mountpoint was the one found in fstab(5), which is
>> not necessarily the one currently mounted.
>
> That's why I didn't do this originally. Changing to /proc/self/mounts is
> a simpler/clearer, but more significant/riskier code change.
>
> If you want to maintain the semantics of "what is configured" rather
> than "what is mounted", then the only way to support ZFS is to call `zfs
> list`. That is how you find out "what is configured" in ZFS.

I'll wait for Jonas' opinion on bypassing /etc/fstab in a FS-specific
manner. In the meantime I'm decreasing the severity to wishlist since
adding ‘initramfs’ to the 4th field of the crypttab(5) entry of each
underlying device is enough to provide root FS encryption for arbitrary
file systems (assuming the initrd can mount it).

--
Guilhem.

[Andras Korn]

Hi,

I saw your discussion about parsing the output of "zpool status" to find out
what devices a zpool needs.

I think parsing the output of "zpool list -PvHp" might be less error-prone.

This is a complex example:

# zpool list -PvHp
backup 1924145348608 1671564828672 252580519936 - 64 86 1.55 ONLINE -
/dev/disk/by-serial/MN1220FA057Y6D_2000TB_hdd-part5 1924145348608 1671564828672 252580519936 - 64 86
cache - - - - - -
/dev/disk/by-serial/S2R6NX0J725133M_250GB_ssd-part5 21337997312 5375048192 15962949120 - 0 25
/dev/disk/by-serial/S2R6NX0J725688Z_250GB_ssd-part5 21337997312 5703546880 15634450432 - 0 26
ssd 35701915648 19918471168 15783444480 - 81 55 1.00 ONLINE -
mirror 35701915648 19918471168 15783444480 - 81 55
/dev/disk/by-id/wwn-0x5002538d421a92fb-part7 - - - - - -
/dev/disk/by-id/wwn-0x5002538d421a95b7-part7 - - - - - -
spare - - - - - -
/dev/disk/by-id/wwn-0x5000cca36ac1fa2d-part4 - - - - - -
/dev/disk/by-id/wwn-0x5000cca36ac232e8-part4 - - - - - -
/dev/disk/by-id/wwn-0x5000cca36ac24219-part4 - - - - - -
/dev/disk/by-id/wwn-0x5000cca36ac263ab-part4 - - - - - -
zroot 1924145348608 1818994868224 105150480384 - 69 94 1.78 ONLINE -
mirror 1924145348608 1818994868224 105150480384 - 69 94
/dev/sdb5 - - - - - -
/dev/sdd5 - - - - - -
/dev/sde5 - - - - - -
log - - - - - -
mirror 503316480 5541888 497774592 - 10 1
/dev/sda3 - - - - - -
/dev/sdc3 - - - - - -
cache - - - - - -
/dev/sda6 42944954368 20906727936 22038226432 - 0 48
/dev/sdc6 42944954368 21442340352 21502614016 - 0 49

Lines with a non-tab at the beginning of the line are pool names or "cache"
or "log" or "spare". Lines that start with a tab introduce member devices
(vdevs), which are either a path to a device or a special string like
"mirror" or "raidz".

To find what devices are needed for the rootfs and swap, you'd read this
list until you find the name of the root pool, then grab the 2nd field of
all lines that start with '\t/' (with TAB as the field separator) until
either EOF or a new pool name is read. This includes spares, but that's as
it should be.

If you also pass in the -L option to 'zpool list', it automatically follows
symlinks in device names.

You can also pass it the name of a pool, and then it'll only list that pool;
e.g.

# zpool list -PvHp gehenna
gehenna 231928233984 169009733632 62918500352 - 59 72 1.00 ONLINE -
/dev/mapper/crypt_sda4 231928233984 169009733632 62918500352 - 59 72

That said, I'm not sure it's worth the effort to support zfs fully
automatically.

I hope this helps.

Best regards,

András

--
The 'S' in 'IoT' stands for Security.